Video question answering method in low-light scenes

Abstract

A video question-answering method for low-light scenarios comprises the steps of video preprocessing, enhancing low-light videos, constructing a VF‑enhNeXt network, enhancing video features, training the VF‑enhNeXt network, verifying the VF‑enhNeXt network, and testing the VF‑enhNeXt network. In the method for enhancing low-light videos, the balance value of the video frame brightness is calculated, and multi-scale enhancement is performed on the video whose balance value meets a set threshold after adjusting the overall brightness of the video frame; in the method for enhancing video features, a VF‑enhNeXt network is constructed, and the VF‑enhNeXt network is used to segment the spatial features of the video and perform spatiotemporal context modeling, thereby obtaining a video question-answering method capable of operating in low-light scenarios. The present invention performs video enhancement and feature enhancement on low-light videos in a video question-answering dataset, provides a multi-level video feature representation for the input of a video question-answering model, and can effectively improve the generalization performance of the video question-answering model in low-light scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of video technology, and particularly relates to a video question-answering method in low-light scenarios. Background Art

[0002] Video question-answering technology refers to a technology that uses a computer to perform reasoning training on a large number of videos and corresponding question-answer pairs described in natural language, constructing a video question-answering model based on a deep neural network, and automatically answering relevant questions described in natural language based on the content of any video segment. Mainstream video question-answering models have demonstrated strong capabilities in cross-modal information fusion and reasoning, and have achieved good results in the field of video question-answering technology. However, it is difficult to effectively obtain useful information and key features in videos of low-light scenes such as underground parking lots and nighttime streets, and the performance of video question-answering methods is poor. At present, although some researchers have proposed low-light video enhancement methods, there are no reports on applying them to video question-answering technology to achieve efficient low-light video question-answering methods or models.

[0003] In the field of video question answering technology, a technical problem that needs to be urgently solved is to provide a video question answering method with high accuracy and wide applicability. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the existing technology and provide a video question answering method in low-light scenes with high accuracy and wide applicability.

[0005] The technical solution adopted to solve the above technical problems consists of the following steps:

[0006] (1) Video preprocessing

[0007] The internationally public benchmark dataset NExT-VQA is used as the video dataset, which is divided into training set, validation set, and test set according to the ratio of 7:1:2. The given video set V is used as input.

[0008] V={v1,v2,…,v n} (1)

[0009] For each input video v n , n is a finite positive integer, sampling a fixed number of video clips P t , P t ∈{P1,P2,...,P t}, t is a finite positive integer, and the video clip P is t Divide into a fixed number of video frames m is a finite positive integer. According to formula (2), the video v n Video frames Darken and add Gaussian noise G:

[0010]

[0011] Among them, I m Represents the preprocessed video frame, α is the darkening coefficient, and its value is [0.4, 0.5].

[0012] (2) Enhanced low-light video

[0013] 1) From the video clip P t Randomly extract the video frame and obtain the balance value of the video frame brightness according to formula (3)

[0014]

[0015] Where N represents the total number of pixels, H, W represent the height and width of the video frame respectively, x, y represent the horizontal and vertical coordinates of the pixel position coordinates in the video frame respectively, R, G, B represent the brightness of the red, green, and blue colors of the image respectively, δ, is the model parameter, δ∈[0.05,0.1],

[0016] 2) Adjust the video v according to formula (4) i Video clip P t Video frame I in m The pixel value I(x,y) at position (x,y):

[0017]

[0018] Among them, (x, y) represents the position coordinates of the pixel in the video frame.

[0019] 3) Video clip P t satisfy T sd is the threshold parameter, and its value is where l max and l min Represent the maximum and minimum values ​​of the video frame brightness respectively. Images of different scales are obtained by repeated downsampling and upsampling. The enhanced image S′ is obtained according to formula (5):

[0020] S′=RL (5)

[0021] R=log(1+S)-H(log(1+S)) (6)

[0022]

[0023] Where L represents the illumination component of the video frame, S is the grayscale image converted from the sampled image, σ represents the variance of the Gaussian function, and x and y represent the coordinates of the center point of the Gaussian function.

[0024] 4) Use the weighted average method to perform scale synthesis on the enhanced image and perform histogram equalization to obtain the enhanced video frame I′ t and video segment P′ t , I′ t ∈{I′1,I′2,...,I′ m}, m is a finite positive integer, P′ t ∈{P′1,P′2,...,P′ t}, t is a finite positive integer.

[0025] (3) Building a VF-enhNeXt network

[0026] The VF-enhNeXt network is composed of a temporal encoding module and a first branch module, a second branch module, an activation function layer, a fully connected layer 3, a feature fusion module, and an inference judgment module connected in series in sequence; the first branch module is composed of a dynamic learning residual block 1 and a fully connected layer 1 connected in series; the second branch module is composed of a dynamic learning residual block 2 and a normalization layer 1, a dynamic learning residual block 3, a normalization layer 2, a dynamic learning residual block 4, and a fully connected layer 2 connected in series in sequence.

[0027] (4) Enhanced video features

[0028] The enhanced video clip P t ′ is passed to the residual convolutional neural network to extract video spatial features and optical flow motion features respectively, and the video features are enhanced through the VF-enhNeXt network as follows:

[0029] 1) Divide the video spatial features into sub-segments along the time dimension and The size of each sub-fragment is M is the batch size, T is the time dimension of the feature, and C is the number of channels. The values ​​of M, T, and C are finite positive integers.

[0030] 2) Use the temporal encoding module to encode the time information of each sub-segment and copy the encoded sub-segments to obtain the same spatiotemporal features and And the spatiotemporal characteristics and As the first branch, spatiotemporal features and As the second branch.

[0031] 3) In the first branch, use the first branch module to connect all the features of the sub-segments along the channel dimension to obtain the first branch result

[0032] 4) In the second branch, use the second branch module to superimpose spatiotemporal features and The same spatiotemporal features in and Concatenate spatiotemporal features along the channel dimension and All the features of the second branch result

[0033] 5) Use Sigmoid function to generate adaptive discrimination weights Where d is the number of input features, and the enhanced feature result F is obtained according to formula (8) a :

[0034]

[0035]

[0036] Where, represents element-by-element addition, ⊙ represents element-by-element multiplication, Sigmoid(·) is the S-type activation function, Conv1d(·) is a one-dimensional convolution, and k is the convolution kernel size.

[0037] (5) Training the VF-enhNeXt network

[0038] 1) Constructing the loss function

[0039] Construct the loss function Ls according to formula (10):

[0040]

[0041] Among them, K represents the number of candidate answers, p w and Represent the scores of incorrect options in the original video sample and the enhanced video sample, p r and Represent the scores of the correct options in the original video sample and the enhanced video sample respectively.

[0042] 2) Training the VF-enhNeXt network

[0043] The video spatial features and optical flow motion features of the training set are fed into the VF-enhNeXt network for training. During the training process, the learning rate is γ∈[10 -5 ,10 -3], the optimizer uses the Adam optimization method and iterates until the loss function converges.

[0044] (6) Verify the VF-enhNeXt network

[0045] The video spatial features and optical flow motion features of the validation set are input into the trained VF-enhNeXt network for verification.

[0046] (7) Testing the VF-enhNeXt network

[0047] Input the video spatial features and optical flow motion features of the test set into the trained VF-enhNeXt network for testing, load the saved weight file, and get the answer.

[0048] In step (3) of constructing the VF-enhNeXt network of the present invention, the temporal encoding module is composed of a convolutional layer, a double-layer temporal attention module, and a normalization layer connected in series, and the convolution kernel size of the convolutional layer is 3×3 and the step size is 1.

[0049] In step (3) of the present invention, the VF-enhNeXt network is constructed, and the dynamic learning residual block is composed of a convolution layer 1 with a convolution kernel size of 1×1 and a step size of 1, a channel-by-channel convolution layer, a convolution layer 2 with a convolution kernel size of 3×3 and a length of 1, an activation function layer, a convolution layer 3 with a convolution kernel size of 1×1 and a length of 1, and a fully connected layer connected in series in sequence; the convolution kernel size of the channel-by-channel convolution layer is 3x3, the step size is 2, the padding number is 1, and the number of groups is the current number of channels.

[0050] In the formula (3) for enhancing low-light video in step (2) of the present invention, the δ, is the model parameter, δ∈[0.05,0.1],

[0051] In the formula (3) for enhancing low-light video in step (2) of the present invention, the δ, are model parameters, The best value is 1.475, and the best value of δ is 0.0526.

[0052] In step (4) of the present invention, in step 1) of enhancing the video features, the video spatial features are divided into sub-segments along the time dimension. and The size of each sub-fragment is M is the batch size, T is the time dimension of the feature, and C is the number of channels. The value of M is 8, the value of T is 16, and the value of C is 3.

[0053] In the step (5) of training the VF-enhNeXt network of the present invention, the training parameter learning rate is γ∈[10 -5 ,10 -3 ], the optimizer uses the Adam optimization method and iterates until the loss function converges.

[0054] In the step (5) of training the VF-enhNeXt network of the present invention, the training parameter learning rate γ is preferably 10 -4 ,The optimizer uses the Adam optimization method and iterates until the loss function converges.

[0055] This method enhances low-light videos and extracts spatial features to obtain a multi-level spatiotemporal feature representation of the video. This method improves the performance of video question-answering tasks and can be used in video question-answering methods in low-light scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flowchart of Example 1 of the present invention.

[0057] Figure 2 This is a structural diagram of the VF-enhNeXt network.

[0058] Figure 3 yes Figure 2 Schematic diagram of the structure of the dynamic learning residual block. DETAILED DESCRIPTION

[0059] The present invention will be further described in detail below with reference to the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0060] Example 1

[0061] exist Figure 1 The video question answering method in low-light scenarios of this embodiment consists of the following steps:

[0062] (1) Video preprocessing

[0063] The internationally public benchmark dataset NExT-VQA is used as the video dataset, which is divided into training set, validation set, and test set according to the ratio of 7:1:2. The given video set V is used as input.

[0064] V={v1,v2,…,v n} (1)

[0065] For each input video v n In this embodiment, n is set to 5440, and a fixed number of video clips P are sampled. t , Pt ∈{P1,P2,...,P t}, in this embodiment, the value of t is 8, and the video segment P is t Divide into a fixed number of video frames In this embodiment, the value of m is 16. According to formula (2), the video v n Video frames Darken and add Gaussian noise G:

[0066]

[0067] Among them, I m represents the preprocessed video frame, α is the darkening coefficient, and the value of α is [0.4, 0.6]. In this embodiment, the value of α is 0.5.

[0068] (2) Enhanced low-light video

[0069] 1) From the video clip P t Randomly extract the video frame and obtain the balance value of the video frame brightness according to formula (3)

[0070]

[0071] Where N represents the total number of pixels, H, W represent the height and width of the video frame respectively, x, y represent the horizontal and vertical coordinates of the pixel position coordinates in the video frame respectively, R, G, B represent the brightness of the red, green, and blue colors of the image respectively, δ, is the model parameter, δ∈[0.05,0.1], The value of δ in this embodiment is 0.0526. The value is 1.475.

[0072] 2) Adjust the video v according to formula (4) i Video clip P t Video frame I in m The pixel value I(x,y) at position (x,y):

[0073]

[0074] Among them, (x, y) represents the position coordinates of the pixel in the video frame.

[0075] 3) Video clip P t satisfy T sd is the threshold parameter, and its value is where l max and l minRepresent the maximum and minimum values ​​of the video frame brightness respectively. Images of different scales are obtained by repeated downsampling and upsampling. The enhanced image S′ is obtained according to formula (5):

[0076] S'=RL (5)

[0077] R=log(1+S)-H(log(1+S)) (6)

[0078]

[0079] Where L represents the illumination component of the video frame, S is the grayscale image converted from the sampled image, σ represents the variance of the Gaussian function, and x and y represent the coordinates of the center point of the Gaussian function.

[0080] 4) The enhanced image is scaled using a weighted averaging method, which has been published in the literature Multi-scale Retinex for color image enhancement (Rahman Z, Jobson DJ, Woodell GA. Multi-scale Retinex for color image enhancement [C]. Proceedings of 3rd IEEE international conference on image processing. IEEE, 1996, 3: 1003-1006.), and histogram equalization is performed to obtain the enhanced video frame I′. t and video segment P′ t , I′ t ∈{I′1,I′2,...,I′ m}, m is a finite positive integer, P′ t ∈{P′1,P′2,...,P′ t}, t is a finite positive integer.

[0081] (3) Building a VF-enhNeXt network

[0082] Figure 2 The structural diagram of the VF-enhNeXt network is given. Figure 2In the embodiment, the VF-enhNeXt network is composed of a temporal coding module, a first branch module, a second branch module, an activation function layer, a fully connected layer 3, a feature fusion module, and an inference judgment module connected in series. The temporal coding module of the embodiment is composed of a convolutional layer, a double-layer temporal attention module, and a normalization layer connected in series. The convolution kernel size of the convolutional layer is 3×3 and the step size is 1. The first branch module of the embodiment is composed of a dynamic learning residual block 1 and a fully connected layer 1 connected in series. The second branch module of the embodiment is composed of a dynamic learning residual block 2 and a normalization layer 1, a dynamic learning residual block 3, a normalization layer 2, a dynamic learning residual block 4, and a fully connected layer 2 connected in series.

[0083] Figure 3 Given Figure 2 Schematic diagram of the structure of the dynamic learning residual block. Figure 3 In the example, the dynamic learning residual block of this embodiment is composed of a convolutional layer 1 with a convolution kernel size of 1×1 and a stride of 1, a channel-by-channel convolutional layer, a convolutional layer 2 with a convolution kernel size of 3×3 and a length of 1, an activation function layer, a convolutional layer 3 with a convolution kernel size of 1×1 and a length of 1, and a fully connected layer, connected in series. The convolution kernel size of the channel-by-channel convolutional layer of this embodiment is 3x3, the stride is 2, the padding number is 1, and the number of groups is the current number of channels.

[0084] (4) Enhanced video features

[0085] The enhanced video clip P t ′ is passed to the residual convolutional neural network to extract video spatial features and optical flow motion features respectively, and the video features are enhanced through the VF-enhNeXt network as follows:

[0086] 1) Divide the video spatial features into sub-segments along the time dimension and The size of each sub-fragment is M is the batch size, T is the time dimension of the feature, and C is the number of channels. In this embodiment, M is 8, T is 16, and C is 3.

[0087] 2) Use the temporal encoding module to encode the time information of each sub-segment and copy the encoded sub-segments to obtain the same spatiotemporal features and And the spatiotemporal characteristics and As the first branch, spatiotemporal features and As the second branch.

[0088] 3) In the first branch, use the first branch module to connect all the features of the sub-segments along the channel dimension to obtain the first branch result

[0089] 4) In the second branch, use the second branch module to superimpose spatiotemporal features and The same spatiotemporal features in and Concatenate spatiotemporal features along the channel dimension and All the features of the second branch result

[0090] 5) Use Sigmoid function to generate adaptive discrimination weights Where d is the number of input features, and the enhanced feature result F is obtained according to formula (8) a .

[0091]

[0092]

[0093] in, represents element-by-element addition, ⊙ represents element-by-element multiplication, Sigmoid(·) is the S-type activation function, Conv1d(·) is a one-dimensional convolution, and k is the convolution kernel size.

[0094] The present invention performs video enhancement on low-light videos and feature enhancement on extracted spatial features of the videos to obtain a multi-level spatiotemporal feature representation of the videos. This invention has the advantage of improving the performance of video question-answering tasks.

[0095] (5) Training the VF-enhNeXt network

[0096] 1) Constructing the loss function

[0097] Construct the loss function Ls according to formula (10):

[0098]

[0099] Among them, K represents the number of candidate answers, p w and Represent the scores of incorrect options in the original video sample and the enhanced video sample, p r and Represent the scores of the correct options in the original video sample and the enhanced video sample respectively.

[0100] 2) Training the VF-enhNeXt network

[0101] The video spatial features and optical flow motion features of the training set are fed into the VF-enhNeXt network for training. During the training process, the learning rate is γ∈[10 -5 ,10-3 ], the value of γ in this embodiment is 10 -4 ,The optimizer uses the Adam optimization method and iterates until the loss function converges.

[0102] (6) Verify the VF-enhNeXt network

[0103] The video spatial features and optical flow motion features of the validation set are input into the trained VF-enhNeXt network for verification.

[0104] (7) Testing the VF-enhNeXt network

[0105] Input the video spatial features and optical flow motion features of the test set into the trained VF-enhNeXt network for testing, load the saved weight file, and get the answer.

[0106] Complete the video question-answering method in low-light scenarios.

[0107] Example 2

[0108] The video question answering method in low-light scenarios of this embodiment comprises the following steps:

[0109] (1) Video preprocessing

[0110] The internationally public benchmark dataset NExT-VQA is used as the video dataset, which is divided into training set, validation set, and test set according to the ratio of 7:1:2. The given video set V is used as input.

[0111] V={v1,v2,…,v n} (1)

[0112] For each input video v n In this embodiment, n is set to 5440, and a fixed number of video clips P are sampled. t , P t ∈{P1,P2,...,P t}, in this embodiment, the value of t is 8, and the video segment P is t Divide into a fixed number of video frames In this embodiment, the value of m is 16. According to formula (2), the video v n Video frames Darken and add Gaussian noise G:

[0113]

[0114] Among them, I m represents the preprocessed video frame, α is the darkening coefficient, and the value of α is [0.4, 0.6]. In this embodiment, the value of α is 0.4.

[0115] (2) Enhanced low-light video

[0116] 1) From the video clip P t Randomly extract the video frame and obtain the balance value of the video frame brightness according to formula (3)

[0117]

[0118] Where N represents the total number of pixels, H, W represent the height and width of the video frame respectively, x, y represent the horizontal and vertical coordinates of the pixel position coordinates in the video frame respectively, R, G, B represent the brightness of the red, green, and blue colors of the image respectively, δ, is the model parameter, δ∈[0.05,0.1], The value of δ in this embodiment is 0.05. The value is 1.2.

[0119] (3) Building a VF-enhNeXt network

[0120] This step is the same as in Example 1.

[0121] (4) Enhanced video features

[0122] This step is the same as in Example 1.

[0123] (5) Training the VF-enhNeXt network

[0124] 1) Constructing the loss function

[0125] This step is the same as in Example 1.

[0126] 2) Training the VF-enhNeXt network

[0127] The video spatial features and optical flow motion features of the training set are fed into the VF-enhNeXt network for training. During the training process, the learning rate is γ∈[10 -5 ,10 -3 ], the value of γ in this embodiment is 10 -5 ,The optimizer uses the Adam optimization method and iterates until the loss function converges.

[0128] The other steps are the same as those in Example 1.

[0129] Complete the video question-answering method in low-light scenarios.

[0130] Example 3

[0131] The video question answering method in low-light scenarios of this embodiment comprises the following steps:

[0132] (1) Video preprocessing

[0133] The internationally public benchmark dataset NExT-VQA is used as the video dataset, which is divided into training set, validation set, and test set according to the ratio of 7:1:2. The given video set V is used as input.

[0134] V={v1,v2,…,v n} (1)

[0135] For each input video v n In this embodiment, n is set to 5440, and a fixed number of video clips P are sampled. t , P t ∈{P1,P2,...,P t}, in this embodiment, the value of t is 8, and the video segment P is t Divide into a fixed number of video frames In this embodiment, the value of m is 16. According to formula (2), the video v n Video frames Darken and add Gaussian noise G:

[0136]

[0137] Among them, I m represents the preprocessed video frame, α is the darkening coefficient, and the value of α is [0.4, 0.6]. In this embodiment, the value of α is 0.6.

[0138] (2) Enhanced low-light video

[0139] 1) From the video clip P t Randomly extract the video frame and obtain the balance value of the video frame brightness according to formula (3)

[0140]

[0141] Where N represents the total number of pixels, H, W represent the height and width of the video frame respectively, x, y represent the horizontal and vertical coordinates of the pixel position coordinates in the video frame respectively, R, G, B represent the brightness of the red, green, and blue colors of the image respectively, δ, is the model parameter, δ∈[0.05,0.1], The value of δ in this embodiment is 0.1. The value is 1.6.

[0142] (3) Building a VF-enhNeXt network

[0143] This step is the same as in Example 1.

[0144] (4) Enhanced video features

[0145] This step is the same as in Example 1.

[0146] (5) Training the VF-enhNeXt network

[0147] 1) Constructing the loss function

[0148] This step is the same as in Example 1.

[0149] 2) Training the VF-enhNeXt network

[0150] The video spatial features and optical flow motion features of the training set are fed into the VF-enhNeXt network for training. During the training process, the learning rate is γ∈[10 -5 ,10 -3 ], the value of γ in this embodiment is 10 -3 ,The optimizer uses the Adam optimization method and iterates until the loss function converges.

[0151] The other steps are the same as those in Example 1.

[0152] Complete the video question-answering method in low-light scenarios.

[0153] In order to verify the beneficial effects of the present invention, the inventors used the video question answering method in low-light scenarios of Example 1 of the present invention (hereinafter referred to as Example 1) and “Xiao J, Shang X, Yao A, et al. Next-qa: Next phase of question-answering to explaining temporal actions[C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 9777-9786.” (hereinafter referred to as Comparative Experiment 1), “Jang Y, Song Y, Yu Y, et al. Tgif-qa: Toward spatio-temporal reasoning in visual question answering[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 2758-2766.” (hereinafter referred to as Comparative Experiment 2), “Gao J, Ge R, Chen K, et al. Motion-appearance co-memory networks for video question answering[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Recognition.2018:6576-6585." (abbreviated as comparative experiment 3) and "Le TM,Le V,Venkatesh S,et al.Hierarchical conditional relation networks for video question answering[C].Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2020:9972-9981." (abbreviated as comparative experiment 4) were compared and the accuracy of the model answering the question ACC was calculated according to formula (11). MC To conduct a comprehensive evaluation:

[0154]

[0155] Among them, Q trepresents the number of question-answer pairs, U represents the size of the question, and a i Indicates the correct answer, p i represents the predicted answer, and I[·] represents an indicator function whose output value is 1 when the two numbers are equal, otherwise it is 0. That is, when the predicted answer is exactly the same as the correct answer, the accuracy is equal to 1, otherwise it is 0.

[0156] The experimental and calculation results are shown in Table 1.

[0157] Table 1 Experimental results of Example 1 and comparative experiments 1-5

[0158] Table 1 Experimental results of Example 1 and comparative experiments 1-5

[0159] Experimental group Accuracy (%) Comparative Experiment 1 44.01 Comparative Experiment 2 43.09 Comparative Experiment 3 43.17 Comparative Experiment 4 43.51 Example 1 45.23

[0160] As can be seen from Table 1, the accuracy of Example 1 is improved by 1.2% over Comparative Experiment 1, 2.2% over Comparative Experiment 2, 2.1% over Comparative Experiment 3, and 1.7% over Comparative Experiment 4.

Claims

1. A video question answering method in low-light scenarios, characterized by It consists of the following steps: (1) Video preprocessing The internationally public benchmark dataset NExT-VQA is used as the video dataset, which is divided into training set, validation set, and test set according to the ratio of 7:1:

2. The given video set V is used as input. V={v1,v2,…,v n } (1) For each input video v n , n is a finite positive integer, sampling a fixed number of video clips P t , P t ∈{P1,P2,...,P t }, t is a finite positive integer, and the video clip P is t Divide into a fixed number of video frames m is a finite positive integer. According to formula (2), the video v n Video frames Darken and add Gaussian noise G: Among them, I m represents the preprocessed video frame, α is the darkening coefficient, and its value is [0.4, 0.5]; (2) Enhanced low-light video 1) From the video clip P t Randomly extract the video frame and obtain the balance value of the video frame brightness according to formula (3) Where N represents the total number of pixels, H, W represent the height and width of the video frame respectively, x, y represent the horizontal and vertical coordinates of the pixel position coordinates in the video frame respectively, R, G, B represent the brightness of the red, green, and blue colors of the image respectively, δ, is the model parameter, δ∈[0.05,0.1], 2) Adjust the video v according to formula (4) i Video clip P t Video frame I in m The pixel value I(x,y) at position (x,y): Where (x, y) represents the position coordinates of the pixel in the video frame; 3) Video clip P t satisfy T sd is the threshold parameter, and its value is where l max and l min Represent the maximum and minimum values ​​of the video frame brightness respectively. Images of different scales are obtained by repeated downsampling and upsampling. The enhanced image S′ is obtained according to formula (5): S′=RL (5) R=log(1+S)-H(log(1+S)) (6) Where L represents the illumination component of the video frame, S is the grayscale image converted from the sampled image, σ represents the variance of the Gaussian function, and x and y represent the coordinates of the center point of the Gaussian function; 4) Use the weighted average method to perform scale synthesis on the enhanced image and perform histogram equalization to obtain the enhanced video frame I′ t and video segment P′ t , I′ t ∈{I′1,I′2,...,I′ m }, m is a finite positive integer, P′ t ∈{P′1,P′2,...,P′ t }, t is a finite positive integer; (3) Building a VF-enhNeXt network The VF-enhNeXt network is composed of a temporal encoding module, a first branch module, a second branch module, an activation function layer, a fully connected layer 3, a feature fusion module, and an inference judgment module connected in series. The first branch module is composed of a dynamic learning residual block 1 and a fully connected layer 1 connected in series. The second branch module is composed of a dynamic learning residual block 2 and a normalization layer 1, a dynamic learning residual block 3, a normalization layer 2, a dynamic learning residual block 4, and a fully connected layer 2 connected in series. (4) Enhanced video features The enhanced video segment P′ t The results are passed to the residual convolutional neural network to extract video spatial features and optical flow motion features respectively, and the video features are enhanced through the VF-enhNeXt network as follows: 1) Divide the video spatial features into sub-segments along the time dimension and The size of each sub-fragment is M is the batch size, T is the time dimension of the feature, C is the number of channels, and M, T, and C are finite positive integers; 2) Use the temporal encoding module to encode the time information of each sub-segment and copy the encoded sub-segments to obtain the same spatiotemporal features and And the spatiotemporal characteristics and As the first branch, spatiotemporal features and As the second branch; 3) In the first branch, use the first branch module to connect all the features of the sub-segments along the channel dimension to obtain the first branch result 4) In the second branch, use the second branch module to superimpose spatiotemporal features and The same spatiotemporal features in and Concatenate spatiotemporal features along the channel dimension and All the features of the second branch result 5) Use Sigmoid function to generate adaptive discrimination weights Where d is the number of input features, and the enhanced feature result F is obtained according to formula (8) a : Where ⊕ represents element-by-element addition, ⊙ represents element-by-element multiplication, Sigmoid(·) is the S-type activation function, Conv1d(·) is the one-dimensional convolution, and k is the convolution kernel size. (5) Training the VF-enhNeXt network 1) Constructing the loss function Construct the loss function Ls according to formula (10): Among them, K represents the number of candidate answers, p w and Represent the scores of incorrect options in the original video sample and the enhanced video sample, p r and Represent the scores of the correct options in the original video sample and the enhanced video sample respectively; 2) Training the VF-enhNeXt network The video spatial features and optical flow motion features of the training set are fed into the VF-enhNeXt network for training. During the training process, the learning rate is γ∈[10 -5 ,10 -3 ], the optimizer uses the Adam optimization method and iterates until the loss function converges; (6) Verify the VF-enhNeXt network The video spatial features and optical flow motion features of the validation set are input into the trained VF-enhNeXt network for verification; (7) Testing the VF-enhNeXt network Input the video spatial features and optical flow motion features of the test set into the trained VF-enhNeXt network for testing, load the saved weight file, and get the answer.

2. The video question-answering method in low-light scenarios according to claim 1, characterized in that: In step (3) of constructing the VF-enhNeXt network, the temporal encoding module is composed of a convolutional layer, a double-layer temporal attention module, and a normalization layer connected in series, and the convolution kernel size of the convolutional layer is 3×3 and the step size is 1.

3. The video question-answering method in low-light scenarios according to claim 1, characterized in that: In step (3) of constructing the VF-enhNeXt network, the dynamic learning residual block is composed of a convolution layer 1 with a convolution kernel size of 1×1 and a step size of 1, a channel-by-channel convolution layer, a convolution layer 2 with a convolution kernel size of 3×3 and a length of 1, an activation function layer, a convolution layer 3 with a convolution kernel size of 1×1 and a length of 1, and a fully connected layer connected in series in sequence; the convolution kernel size of the channel-by-channel convolution layer is 3x3, the step size is 2, the padding number is 1, and the number of groups is the current number of channels.

4. The video question-answering method in low-light scenarios according to claim 1, characterized in that: In the formula (3) for enhancing low-light video in step (2), the δ, is the model parameter, δ∈[0.05,0.1], 5. The video question-answering method in low-light scenarios according to claim 1 or 4, characterized in that: In the formula (3) for enhancing low-light video in step (2), the δ, are model parameters, The value is 1.475 and the value of δ is 0.0526.

6. The video question-answering method in low-light scenarios according to claim 1, characterized in that: In step (4) of enhancing video features, the video spatial features are divided into sub-segments along the time dimension. and The size of each sub-fragment is M is the batch size, T is the time dimension of the feature, and C is the number of channels. The value of M is 8, the value of T is 16, and the value of C is 3.

7. The video question-answering method in low-light scenarios according to claim 1, characterized in that: In step (5) of training the VF-enhNeXt network, the training parameter learning rate is γ∈[10 -5 ,10 -3 ], the optimizer uses the Adam optimization method and iterates until the loss function converges.

8. The video question-answering method in low-light scenarios according to claim 1 or 7, characterized in that: In step (5) of training the VF-enhNeXt network, the training parameter learning rate γ is 10 -4 ,The optimizer uses the Adam optimization method and iterates until the loss function converges.

Images