Large-scale video action recognition method based on Fourier transform saliency

Abstract

The embodiments of the present disclosure disclose a large-scale video behavior recognition method based on Fourier transform saliency. A specific implementation of the method includes: generating a group of to-be-processed video frames corresponding to the to-be-processed video data; generating each frequency domain transformation matrix corresponding to the to-be-processed video frame group; determining a target video frame corresponding to the to-be-processed video data; generating target question text data; performing masking processing on the target question text data to obtain masked question text data; inputting the target video frame into the image processing layer of a pre-trained behavior recognition model; inputting the masked question text data into the text processing layer of the behavior recognition model; inputting the video frame feature information and the video text feature information into the output layer of the behavior recognition model to obtain behavior description information; and sending the behavior description information to the user terminal. This implementation reduces the computing resources consumed during recognition, which can reduce the waste of computing resources.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a large-scale video behavior recognition method based on Fourier transform saliency. Background Art

[0002] Video behavior recognition is a technology that identifies behaviors in videos. Currently, the typical approach for identifying behaviors in videos is to first extract features from each video frame using a feature extraction algorithm to generate a corresponding feature vector. A pre-trained classifier is then used to classify the feature vectors and obtain a description of the corresponding behavior.

[0003] However, when using the above method to identify behaviors in videos, the following technical problems often arise:

[0004] When identifying behaviors in videos, the recognition process is easily affected by background information, resulting in low recognition accuracy. This in turn increases the probability of needing to re-identify the video, which wastes computing resources. Furthermore, when using feature extraction algorithms to extract features from videos, every frame must be processed, consuming a large amount of computing resources. Summary of the Invention

[0005] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure propose a large-scale video behavior recognition method based on Fourier transform saliency to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In the first aspect, some embodiments of the present disclosure provide a large-scale video behavior recognition method based on Fourier transform saliency, the method comprising: in response to receiving the to-be-processed video data and question text data sent by the user terminal, generating a to-be-processed video frame group corresponding to the above-mentioned to-be-processed video frame group; based on the above-mentioned to-be-processed video frame group, generating each frequency domain transformation matrix corresponding to the above-mentioned to-be-processed video frame group; based on the above-mentioned each frequency domain transformation matrix, determining the target video frame corresponding to the above-mentioned to-be-processed video data; based on the above-mentioned question text data, generating target question text data; performing mask processing on the above-mentioned target question text data to obtain a masked question text. This data; input the above-mentioned target video frame into the image processing layer of the pre-trained behavior recognition model to obtain video frame feature information, wherein the above-mentioned image processing layer includes an image mapping network, an image embedding network and an image linear layer; input the above-mentioned masked question text data into the text processing layer of the above-mentioned behavior recognition model to obtain video text feature information, wherein the above-mentioned text processing layer includes a text mapping network, a text embedding layer and a text linear layer; input the above-mentioned video frame feature information and the above-mentioned video text feature information into the output layer of the above-mentioned behavior recognition model to obtain behavior description information; and send the above-mentioned behavior description information to the above-mentioned user terminal.

[0008] In the second aspect, some embodiments of the present disclosure provide a large-scale video behavior recognition device based on Fourier transform saliency, the device comprising: a first generating unit, configured to generate a group of to-be-processed video frames corresponding to the above-mentioned to-be-processed video data in response to receiving the to-be-processed video data and question text data sent by the user terminal; a second generating unit, configured to generate each frequency domain transformation matrix corresponding to the above-mentioned to-be-processed video frame group based on the above-mentioned to-be-processed video frame group; a determining unit, configured to determine the target video frame corresponding to the above-mentioned to-be-processed video data based on the above-mentioned each frequency domain transformation matrix; a third generating unit, configured to generate target question text data based on the above-mentioned question text data; a masking unit, configured to perform masking on the above-mentioned target question text data to obtain a masked image. The first input unit is configured to input the above-mentioned masked question text data into the image processing layer of the pre-trained behavior recognition model to obtain video frame feature information, wherein the above-mentioned image processing layer includes an image mapping network, an image embedding network and an image linear layer; the second input unit is configured to input the above-mentioned masked question text data into the text processing layer of the above-mentioned behavior recognition model to obtain video text feature information, wherein the above-mentioned text processing layer includes a text mapping network, a text embedding layer and a text linear layer; the third input unit is configured to input the above-mentioned video frame feature information and the above-mentioned video text feature information into the output layer of the above-mentioned behavior recognition model to obtain behavior description information; the sending unit is configured to send the above-mentioned behavior description information to the above-mentioned user terminal.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0011] The above-described embodiments of the present disclosure have the following beneficial effects: The large-scale video action recognition method based on Fourier transform saliency in some embodiments of the present disclosure can reduce the waste of computing resources. Specifically, the reason for this waste of computing resources is that when identifying actions in a video, the recognition process is easily interfered with by background information in the video, resulting in low recognition accuracy. This in turn leads to a high probability of requiring computing resources to re-identify the video, which in turn leads to a waste of computing resources. Furthermore, when using a feature extraction algorithm to extract features from a video, each frame in the video must be processed, resulting in a high consumption of computing resources. Based on this, the large-scale video action recognition method based on Fourier transform saliency in some embodiments of the present disclosure first generates a set of to-be-processed video frames corresponding to the to-be-processed video data in response to receiving to-be-processed video data and question text data sent by a user terminal. This generates the video data to be identified. Secondly, based on the to-be-processed video frame set, frequency domain transform matrices corresponding to the to-be-processed video frame set are generated. Thus, by performing Fourier transform processing on the video, the model can better identify foreground information during recognition and reduce interference from background information. Then, based on each of the frequency domain transformation matrices, a target video frame corresponding to the video data to be processed is determined. This allows the most significant video frame to be processed to be determined from the group of video frames to be processed. Then, based on the question text data, target question text data is generated. Then, the target question text data is masked to obtain masked question text data. This allows redundant information in the text data to be masked. Then, the target video frame is input into the image processing layer of a pre-trained behavior recognition model to obtain video frame feature information. The image processing layer includes an image mapping network, an image embedding network, and an image linearization layer. Thus, the target video frame can be processed using the behavior recognition model. Then, the masked question text data is input into the text processing layer of the behavior recognition model to obtain video text feature information. The text processing layer includes a text mapping network, a text embedding layer, and a text linearization layer. Thus, the masked question text data can be processed using the behavior recognition model. Then, the video frame feature information and the video text feature information are input into the output layer of the behavior recognition model to obtain behavior description information. Thus, behavior description information corresponding to the video can be obtained. Finally, the behavior description information is sent to the user terminal. Furthermore, because the most significant video frame to be processed can be identified from the group of video frames to be processed, and then processed to obtain the behavior description information, it is not necessary to process every frame in the video data to be processed. This reduces the computing resources consumed during processing.In addition, because the video can be processed through Fourier transform first, the model can better process foreground information and reduce the interference of background information on recognition, thereby improving the recognition accuracy and reducing the probability of needing to consume computing resources to re-recognize the video, thereby reducing the waste of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of the large-scale video behavior recognition method based on Fourier transform saliency according to the present disclosure;

[0014] Figure 2 1 is a schematic structural diagram of some embodiments of a large-scale video behavior recognition device based on Fourier transform saliency according to the present disclosure;

[0015] Figure 3 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] Figure 1 The process 100 of some embodiments of the large-scale video behavior recognition method based on Fourier transform saliency according to the present disclosure is shown. The large-scale video behavior recognition method based on Fourier transform saliency includes the following steps:

[0023] Step 101: In response to receiving the to-be-processed video data and question text data sent by the user terminal, a to-be-processed video frame group corresponding to the to-be-processed video data is generated.

[0024] In some embodiments, an execution entity (e.g., a computing device) of a large-scale video action recognition method based on Fourier transform saliency may generate a set of to-be-processed video frames corresponding to the to-be-processed video data in response to receiving to-be-processed video data and question text data sent by a user terminal. The to-be-processed video data may be the video data to be recognized. The question text data may be a question posed by a user based on the to-be-processed video data. For example, the question text data may be "What is the person in the video doing?" Each to-be-processed video frame in the to-be-processed video frame set may be a video frame in the to-be-processed video data. Each to-be-processed video frame in the to-be-processed video frame set may be an RGB image. Each pixel in the to-be-processed video frame may have a corresponding red, green, and blue color channel value. The red color channel value may be the color channel value corresponding to the R channel in the RGB image. The green color channel value may be the color channel value corresponding to the G channel in the RGB image. The blue color channel value may be the color channel value corresponding to the B channel in the RGB image. Each to-be-processed video frame in the to-be-processed video frame set may have corresponding image height data and image width data. The image height data may be the image height corresponding to the video frame to be processed. The image width data may be the image width corresponding to the video frame to be processed. The execution entity may be a server. In practice, the execution entity may determine each video frame in the video data to be processed as a video frame group to be processed.

[0025] Step 102: Generate frequency domain transformation matrices corresponding to the video frame group to be processed based on the video frame group to be processed.

[0026] In some embodiments, the execution entity may generate frequency domain transformation matrices corresponding to the video frame group to be processed based on the video frame group to be processed, wherein each of the frequency domain transformation matrices may be a complex matrix used to represent the frequency domain of the corresponding video frame to be processed.

[0027] In some optional implementations of some embodiments, the execution entity may generate frequency domain transformation matrices corresponding to the video frame group to be processed based on the video frame group to be processed by the following steps:

[0028] For each video frame to be processed in the above group of video frames to be processed, perform the following steps:

[0029] In the first step, image conversion processing is performed on the video frame to be processed based on a preset first gamma value, a preset second gamma value, and a preset third gamma value to obtain grayscale video frame data. The first gamma value may be a value used to convert the red color channel value of each element in the video frame to be processed. For example, the first gamma value may be 0.299. The second gamma value may be a value used to convert the green color channel value of each element in the video frame to be processed. For example, the second gamma value may be 0.587. The third gamma value may be a value used to convert the blue color channel value of each element in the video frame to be processed. For example, the blue color channel value may be 0.114. The grayscale video frame data may be a grayscale image corresponding to the video frame to be processed. In practice, for each pixel in the video frame to be processed, the execution entity may first determine the first channel value as the product of the red color channel value corresponding to the pixel and the first gamma value. Then, the product of the green color channel value corresponding to the above pixel point and the above second grayscale coefficient value can be determined as the second channel value. Then, the product of the blue color channel value corresponding to the above pixel point and the above third grayscale coefficient value can be determined as the third channel value. Then, the sum of the above first channel value, the above second channel value and the above third channel value can be determined as the grayscale value corresponding to the above pixel point. Then, according to the corresponding positions of the above pixel points in the above video frame to be processed, the determined grayscale values ​​can be combined into a matrix as a grayscale matrix. Finally, the above grayscale matrix can be rendered by an image rendering function to obtain grayscale video frame data. Among them, the above image rendering function can be a function that can render pixel values ​​into an image. For example, the above image rendering function can be an imshow function.

[0030] The second step is to perform frequency domain transformation on the grayscale video frame data to obtain a frequency domain transformation matrix.

[0031] In some optional implementations of some embodiments, the execution entity may perform frequency domain transformation processing on the grayscale video frame data through the following steps to obtain a frequency domain transformation matrix:

[0032] The first step is to determine each matrix height data and each matrix width data based on the image height data and image width data corresponding to the above-mentioned video frame to be processed. Each matrix height data in the above-mentioned matrix height data may be a non-negative integer less than the above-mentioned image height data. Each matrix width data in the above-mentioned matrix width data may be a non-negative integer less than the above-mentioned image width data. In practice, the above-mentioned execution entity may determine each non-negative integer less than the above-mentioned image height data as each matrix height data. Each non-negative integer less than the above-mentioned image width data may be determined as each matrix width data.

[0033] The second step is to determine a grayscale video matrix based on the grayscale video frame data. The grayscale video matrix may be an image matrix corresponding to the grayscale video frame data. In practice, the execution entity may determine the image matrix corresponding to the grayscale video frame data as the grayscale video matrix.

[0034] In the third step, for each matrix height data in the above matrix height data and each matrix width data in the above matrix width data, the following steps are performed:

[0035] The first sub-step is to perform the following steps for each element of the grayscale video matrix:

[0036] Sub-step one, based on the above-mentioned grayscale video matrix and the above-mentioned elements, determine the element height data and element width data corresponding to the above-mentioned elements. Among them, the above-mentioned element height data can be the height corresponding to the above-mentioned element in the above-mentioned grayscale video matrix. The above-mentioned element width data can be the width corresponding to the above-mentioned element in the above-mentioned grayscale video matrix. In practice, the above-mentioned execution entity can determine the image height of the above-mentioned element in the above-mentioned grayscale video matrix as the element height. Then, the difference between the above-mentioned element height and the preset value can be determined as the element height data. Then, the image width corresponding to the above-mentioned element in the above-mentioned grayscale video matrix can be determined as the element width. Then, the difference between the above-mentioned element width and the above-mentioned preset value can be determined as the element width data. Among them, the above-mentioned preset value can be 1.

[0037] Sub-step 2: determining the product of the matrix height data and the element height data as the matrix element height data.

[0038] Sub-step three: determining the product of the matrix width data and the element width data as the matrix element width data.

[0039] Sub-step four: determining the ratio of the matrix element height data to the image height data corresponding to the video frame to be processed as height ratio data.

[0040] Sub-step five: determining the ratio of the matrix element width data to the image width data corresponding to the video frame to be processed as width ratio data.

[0041] Sub-step six: determining the sum of the height ratio data and the width ratio data as the sum ratio data.

[0042] Sub-step seven: Determine the product of the preset imaginary data, the preset coefficient data, and the summed ratio data as exponential data. The imaginary data may be an imaginary unit. The coefficient data may be the product of pi and a preset multiple. The preset multiple may be a pre-set value. For example, the preset multiple may be 2.

[0043] Sub-step eight: Generate exponential data to be processed based on preset exponential basis data and the exponential data. The exponential basis data may be a natural constant e. In practice, the execution entity may determine the exponential data raised to the power of the exponential basis data as the exponential data to be processed. For example, when the exponential basis data is the natural constant e and the exponential data is 2πi, the exponential data to be processed may be e raised to the power of 2πi. Here, i may represent the imaginary data.

[0044] Sub-step nine: determining the product of the element value corresponding to the above element and the above index data to be processed as the matrix sub-element data.

[0045] The second sub-step is to determine the sum of the determined matrix sub-element data as the matrix element data corresponding to the above-mentioned matrix height data and the above-mentioned matrix width data.

[0046] The fourth step is to combine the obtained matrix element data into a frequency domain transformation matrix. In practice, the execution entity may combine the above matrix element data into a frequency domain transformation matrix based on the matrix height data and matrix width data corresponding to each matrix element data. For example, when the matrix height data corresponding to the matrix element data is 3 and the corresponding matrix width data is 2, the matrix element data should be arranged in the 3rd row and 2nd column of the frequency domain transformation matrix.

[0047] Step 103: Determine a target video frame corresponding to the video data to be processed based on each frequency domain transformation matrix.

[0048] In some embodiments, the execution entity may determine a target video frame corresponding to the video data to be processed based on the frequency domain transformation matrices. The target video frame may be a video frame to be processed for behavior recognition selected from the group of video frames to be processed.

[0049] In some optional implementations of some embodiments, the execution entity may determine the target video frame corresponding to the video data to be processed based on the frequency domain transformation matrices through the following steps:

[0050] In the first step, for each of the above frequency domain transformation matrices, perform the following steps:

[0051] The first sub-step is to generate a spectrum energy matrix based on the frequency domain transformation matrix. The spectrum energy matrix can be a real number matrix used to characterize the energy distribution of the corresponding video frame to be processed in the frequency domain. In practice, first, the execution subject can input the frequency domain transformation matrix into the absolute value function to obtain the matrix output by the absolute value function as the amplitude matrix. The absolute value function can be a function that can generate the amplitude spectrum of a complex matrix. For example, the absolute value function can be an abs function. Then, the logarithm of the amplitude matrix can be determined as the amplitude logarithm matrix. Finally, the product of the preset energy coefficient and the amplitude logarithm matrix can be determined as the spectrum energy matrix. The energy coefficient can be a pre-set value. Here, there is no limitation on the specific setting of the energy coefficient. For example, the energy coefficient can be 0.9.

[0052] The second sub-step is to generate spectral element cumulative data based on the spectral energy matrix. The spectral element cumulative data may be an average value corresponding to each element value in the spectral energy matrix. In practice, the execution entity may determine the average value corresponding to each element value included in the spectral energy matrix as the spectral element cumulative data.

[0053] In the third sub-step, the product of the image height data and the image width data corresponding to the video frame to be processed is determined as image processing data.

[0054] The fourth sub-step is to determine the ratio of the spectrum element accumulated data to the image processed data as the average energy data.

[0055] In the second step, the average energy data that satisfies a preset energy data condition among the determined average energy data is determined as the target average energy data, wherein the preset energy data condition may be that the average energy data has the largest value among the average energy data.

[0056] In the third step, the to-be-processed video frame corresponding to the target average energy data in the to-be-processed video frame group is determined as the target video frame.

[0057] Step 104: Generate target question text data based on the question text data.

[0058] In some embodiments, the execution entity may generate target question text data based on the question text data. The target question text data may be text data after the question text data is sorted. In practice, the execution entity may add preset user text to the question text data to obtain target question text data. The user text may be "asker: user". For example, when the question text data is "What are the people in the video doing?" and the user text is "asker: user", the generated target question text data may be "What are the people in the video doing?; questioner: user".

[0059] Step 105: Mask the target question text data to obtain masked question text data.

[0060] In some embodiments, the execution subject may perform masking on the target question text data to obtain masked question text data. The masked question text data may be text data obtained by masking some contents in the target question text data. In practice, first, the execution subject may use a search instruction to search for each character in the target question text data that meets a preset replacement condition as each character to be replaced. Then, a replacement instruction may be used to replace each character to be replaced in the target question text data with a preset mask character to obtain masked question text data. The mask character may be [MASK]. The preset replacement condition may be that the character exists in a preset replacement character group. Each replacement character in the replacement character group may be a character pre-set by a technician and needs to be replaced with the mask character. For example, the replacement character group may include but is not limited to: "this", "that", and "what".

[0061] Step 106: Input the target video frame into the image processing layer of the pre-trained behavior recognition model to obtain video frame feature information.

[0062] In some embodiments, the execution entity may input the target video frame into the image processing layer of a pre-trained behavior recognition model to obtain video frame feature information. The video frame feature information may be a linearly transformed feature vector corresponding to the target video frame. The behavior recognition model may be a neural network model that takes the target video frame and masked question text data as input and outputs behavior description information. The behavior recognition model may include three layers.

[0063] The first layer may be an image processing layer, wherein the image processing layer may include an image mapping network, an image embedding network, and an image linear layer.

[0064] The image mapping network may be a neural network that takes a target video frame as input and outputs video frame information corresponding to the target video frame. For example, the image mapping network may be a convolutional neural network. The video frame information may be a feature vector corresponding to the target video frame.

[0065] The image embedding network can be a neural network that takes video frame information corresponding to the target video frame as input and outputs a weighted matrix corresponding to the target video frame. For example, the neural network can be a Transformer. The image embedding network corresponds to a query weight matrix, a key weight matrix, and a value weight matrix.

[0066] In practice, first, the image embedding network may determine the product of the query weight matrix and the video frame information as a query matrix. Second, the image embedding network may determine the product of the key weight matrix and the video frame information as a key matrix. Then, the image embedding network may determine the product of the value weight matrix and the video frame information as a value matrix. Then, the image embedding network may determine the transpose of the key matrix as a key transpose matrix. Then, the product of the query matrix and the key transpose matrix may be determined as a product matrix. Then, the dimension of the video frame information may be determined as feature dimension data. Then, the square root of the feature dimension data may be determined as dimension square root data. Then, the ratio of the product matrix to the dimension square root data may be determined as dimension ratio data. Then, the dimension ratio data may be input into a normalization function to obtain the data output by the normalization function as attention score data. Finally, the product of the attention score data and the value matrix may be determined as a weight matrix. The query weight matrix may be a weight matrix used to generate the query matrix. The key weight matrix may be a weight matrix used to generate the key matrix. The value weight matrix may be a weight matrix used to generate the value matrix. The normalization function may be a Softmax function.

[0067] The image linear layer may be a linear layer that takes a weighted matrix corresponding to a target video frame as input and outputs video frame feature information corresponding to the target video frame. The image linear layer may correspond to a linear matrix and an image bias vector.

[0068] In practice, first, the image linear layer can determine the product of the linear matrix and the weighting matrix as a linear weighting matrix. Then, each column element in the linear weighting matrix can be added to the image bias vector to obtain the added matrix as the video frame feature information. Among them, the linear matrix can be a matrix that can be left-multiplied by the weighting matrix and used to map the weighting matrix to a preset feature dimension. As an example, when the feature dimension to be mapped is 3-dimensional and the weighting matrix is ​​a matrix with 2 rows and 2 columns, the number of rows of the linear matrix corresponds to the feature dimension, and the number of columns of the linear matrix is ​​the number of rows of the weighting matrix, that is, the linear matrix is ​​a 3-row, 2-column matrix. The image bias vector can be an array with the same number of rows as the number of rows of the linear weighting matrix and the corresponding number of columns as 1.

[0069] The second layer may be a text processing layer. The text processing layer may include a text mapping network, a text embedding layer, and a text linear layer. The text mapping network may be a neural network that takes masked question text data as input and outputs question text feature information corresponding to the masked question text data. For example, the text mapping network may be a recurrent neural network. The question text feature information may be a feature vector corresponding to the masked question text data.

[0070] The text embedding layer may be a fully connected layer that takes question text feature information corresponding to the masked question text data as input and outputs text embedding feature information corresponding to the masked question text data. The text embedding feature information may be question text feature information processed by the text embedding layer. The text embedding layer may have a pre-trained weight matrix.

[0071] In practice, the text embedding layer can first fine-tune the pre-trained weight matrix using parameter fine-tuning techniques, obtaining the fine-tuned pre-trained weight matrix as the processing matrix. The text embedding layer can then determine the product of the processing matrix and the question text feature information as the text embedding feature information. The parameter fine-tuning technique can be a technique capable of fine-tuning the parameters of a neural network. For example, the parameter fine-tuning technique can be LoRA (Low-Rank Adaptation). The pre-trained weight matrix can be the parameter matrix used in the text embedding layer to process the question text feature information.

[0072] The text linear layer may be a linear layer that takes text embedding feature information corresponding to the masked question text data as input and outputs video text feature information corresponding to the masked question text data. The video text feature information may be text embedding feature information that has undergone a linear transformation. The text linear layer corresponds to a text weight matrix.

[0073] In practice, first, the above-mentioned text linear layer can determine the product of the text weight matrix and the above-mentioned text embedding feature information as a text linear matrix. Then, each column element in the above-mentioned text linear matrix can be added to the preset text bias vector to obtain the added matrix as the video text feature information. Among them, the text weight matrix can be a parameter matrix that can be multiplied by the text embedding feature information on the left and used to map the text embedding feature information to the above-mentioned feature dimension. The above-mentioned text bias vector can be an array with the same number of rows as the number of rows of the text linear matrix and the corresponding number of columns as 1.

[0074] The third layer may be an output layer. The output layer may include a normalized activation function and an output function. The normalized activation function may be a Softmax function that takes the video frame feature information output by the image linear layer and the video text feature information output by the text linear layer as input, and outputs behavior probability distribution data. The behavior probability distribution data may be a probability distribution for characterizing the behavior corresponding to the target video frame. The behavior probability distribution data may include behavior information and behavior probability. The behavior information may be used to describe the behavior corresponding to the target video frame. For example, the behavior information may be "running." The behavior probability may be the accuracy rate corresponding to the behavior information. For example, the behavior probability distribution data may be {"behavior information: running; behavior probability: 70%", "behavior information: walking; behavior probability: 30%"}.

[0075] The output function may be an argmax function that takes the behavior probability distribution data output by the normalized activation function as input and the behavior description information as output. The behavior description information may be the behavior information with the highest corresponding behavior probability in the behavior probability distribution data.

[0076] In some optional implementations of some embodiments, the behavior recognition model may be obtained by training the execution subject through the following steps:

[0077] The first step is to obtain a sample set. Each sample in the sample set includes a sample video frame, sample masked question text data, and sample behavior description information. The sample video frame may be a target video frame used for model training. The sample masked question text data may be masked question text data used for model training. The sample behavior description information may be standard behavior description information corresponding to the sample video frame and the sample masked question text data. The sample behavior description information includes various sample tokens. Each of the sample tokens may be a token in the sample behavior description information.

[0078] In the second step, based on the sample set, the following training steps are performed:

[0079] In the first sub-step, each sample video frame included in at least one sample in the sample set is input into the image mapping network of the image processing layer of the initial neural network to obtain information about each sample video frame. Each sample video frame information in the aforementioned sample video frame information may be video frame information corresponding to the sample video frame. Each sample video frame information in the aforementioned sample video frame information corresponds to a sample. The structure of the initial neural network can be referenced to the aforementioned behavior recognition model and will not be further described here.

[0080] In a second sub-step, the sample video frame information corresponding to the at least one sample is input into the image embedding network in the image processing layer of the initial neural network to obtain sample weight matrices. Each of the sample weight matrices may be a weight matrix corresponding to a sample video frame. Each of the sample weight matrices corresponds to one sample.

[0081] In a third sub-step, each sample weighting matrix corresponding to the at least one sample is input into an image linear layer within the image processing layer of the initial neural network to obtain feature information of each sample video frame corresponding to the at least one sample. Each sample video frame feature information in the sample video frame feature information may be video frame feature information corresponding to the sample video frame.

[0082] In a fourth sub-step, the masked question text data of each sample included in the at least one sample is input into a text mapping network in a text processing layer of the initial neural network to obtain feature information of each sample question text. Each sample question text feature information in the sample question text feature information may be feature information of the question text corresponding to the sample video frame. Each sample question text feature information in the sample question text feature information corresponds to one sample.

[0083] In a fifth sub-step, the sample question text feature information corresponding to the at least one sample is input into a text embedding layer within the text processing layer of the initial neural network to obtain text embedding feature information for each sample. Each sample text embedding feature information in the above-mentioned sample text embedding feature information may be text embedding feature information corresponding to a sample video frame. Each sample text embedding feature information in the above-mentioned sample text embedding feature information corresponds to one sample.

[0084] In a sixth sub-step, the sample text embedding feature information corresponding to the at least one sample is input into the text linear layer of the text processing layer of the initial neural network to obtain the sample video text feature information. Each sample video text feature information in the sample video text feature information may be video text feature information corresponding to a sample video frame. Each sample video text feature information in the sample video text feature information corresponds to one sample.

[0085] In a seventh sub-step, for each of the at least one sample, alignment loss data corresponding to the sample is generated based on the sample video frame feature information and the sample video text feature information. The alignment loss data may be a loss value corresponding to the sample.

[0086] In practice, first, each sample video frame feature information included in the at least one sample can be determined as each target video frame feature information. Secondly, for each sample in the at least one sample, based on the sample video text feature information corresponding to the sample, the feature similarity between the sample video text feature information and each target video frame feature information in the target video frame feature information can be determined as a feature similarity, thereby obtaining a respective feature similarity. Then, for each of the feature similarities, a predetermined exponential basis data corresponding to the power of the feature similarity can be determined as feature index data. The exponential basis data can be a natural constant e. As an example, when the feature similarity is 3, the corresponding feature index data is e raised to the power of 3. Then, the sum of the determined feature index data can be determined as comprehensive index data. Furthermore, the feature index data corresponding to the sample in the respective feature index data can be determined as target feature index data. As an example, the feature index data corresponding to the sample is the feature index data generated from the sample video frame feature information and the sample video text feature information included in the sample. Then, the ratio of the target feature index data to the comprehensive index data can be determined as index ratio data. Finally, the negative logarithm of the exponential ratio data may be determined as the alignment loss data corresponding to the sample.

[0087] The eighth sub-step is, in response to determining that the generated alignment loss data do not meet the preset loss conditions, adjusting the network parameters of the image processing layer and the text processing layer in the initial neural network, and using unused samples to form a sample set, and using the adjusted image processing layer and text processing layer of the initial neural network to perform the above training steps again.

[0088] The preset loss condition may be that the average value of each alignment loss data is less than a preset loss value. The preset loss value may be a pre-set value. No limitation is imposed on the specific setting of the preset loss value. The network parameters corresponding to the image mapping network in the image processing layer and the text mapping network in the text processing layer are frozen, that is, when the network parameters of the initial neural network are adjusted, the network parameters of the image mapping network in the image processing layer and the text mapping network in the text processing layer remain unchanged.

[0089] In practice, first, the above-mentioned execution entity can determine the average value of the above-mentioned individual alignment loss data as the loss value to be processed. Then, for the query weight matrix corresponding to the image embedding network in the image processing layer in the initial neural network, first, the partial derivative of the above-mentioned loss value to be processed and the query weight matrix can be determined as gradient data. Then, the product of the preset learning rate, the above-mentioned gradient data and the above-mentioned loss value to be processed can be determined as the query weight matrix to update the query weight matrix. For the key weight matrix and value weight matrix corresponding to the image embedding network in the image processing layer in the initial neural network, as well as the linear matrix in the image linear layer, the key weight matrix, the value weight matrix and the linear matrix can be updated. Among them, the method for updating the key weight matrix, the value weight matrix and the linear matrix can refer to the specific implementation method for updating the query weight matrix, which will not be repeated here. In this way, the network parameters of the image processing layer in the initial neural network can be adjusted. In practice, the above-mentioned execution entity can adjust the network parameters of the text processing layer in the initial neural network through the backpropagation algorithm.

[0090] Optionally, after generating alignment loss data corresponding to each of the at least one sample based on the sample video frame feature information and the sample video text feature information, the execution entity may further perform the following steps:

[0091] In response to determining that each of the generated alignment loss data satisfies a preset loss condition, the following steps are performed:

[0092] In the first step, the feature information of each sample video frame and the feature information of each sample video text corresponding to the at least one sample are input into the output layer of the initial neural network to obtain each behavior description information corresponding to the at least one sample. Each of the behavior description information includes each word unit. Each word unit in the word unit can be a word unit in the behavior description information.

[0093] In a second step, for each of the at least one sample, text indicator data corresponding to the sample is generated based on the behavior description information corresponding to the sample in the individual behavior description information and the sample behavior description information included in the sample. The text indicator data may be a numerical value used to indicate the accuracy of the generated behavior description information.

[0094] In the third step, in response to determining that the generated text indicator data satisfies a preset indicator data condition, the initial neural network is determined to be a behavior recognition model. The indicator data condition may be that the average value of the text indicator data is greater than a preset indicator value. The preset indicator value may be a pre-set value. The specific setting of the preset indicator value is not limited herein.

[0095] In step 4, in response to determining that the generated text index data does not meet the above index data conditions, the network parameters of the output layer of the initial neural network are adjusted, and the sample set is composed of unused samples, and the above training steps are performed again using the output layer of the adjusted initial neural network. In practice, the network parameters of the output layer of the initial neural network can be adjusted using a backpropagation algorithm.

[0096] In the process of adopting technical solutions to solve the above technical problems, the following problems often arise:

[0097] When using feature extraction algorithms and classifiers to identify behaviors in videos, it is easily interfered with by background information in the video, resulting in low recognition accuracy, which in turn leads to the waste of computing resources when re-identifying the identified videos and consuming computing resources again.

[0098] Faced with the above technical problems, we decided to adopt the following solutions:

[0099] In some optional implementations of some embodiments, the execution entity may generate text indicator data corresponding to the sample based on the behavior description information corresponding to the sample in each behavior description information and the sample behavior description information included in the sample through the following steps:

[0100] The first step is to determine the word units included in the behavior description information as a word unit sequence. In practice, the execution entity can arrange the word units into a word unit sequence according to the order of the word units in the behavior description information, from front to back.

[0101] The second step is to determine the sample tokens included in the sample behavior description information as a sample token sequence. In practice, the execution entity may arrange the sample tokens into a sample token sequence in the order of their appearance in the sample behavior description information, from front to back.

[0102] In the third step, for each of the preset word unit numbers, perform the following steps:

[0103] The first sub-step is to generate each word-gram subsequence based on the number of word-grams and the word-gram sequence. Each word-gram subsequence in each word-gram subsequence may be a subsequence in the word-gram sequence. In practice, for each adjacent word-gram of the number of word-grams in the word-gram sequence, the execution entity may determine the number of word-grams as a word-gram subsequence, thereby obtaining each word-gram subsequence.

[0104] The second sub-step is to generate sample word-gram subsequences based on the number of word-grams and the sample word-gram sequence. Each sample word-gram subsequence in each sample word-gram subsequence may be a subsequence of the sample word-gram sequence. In practice, for each adjacent number of sample word-grams in the sample word-gram sequence, the number of sample word-grams may be determined as a sample word-gram subsequence, thereby obtaining each sample word-gram subsequence.

[0105] In the third sub-step, for each word-unit sub-sequence in the above-mentioned word-unit sub-sequences, perform the following steps:

[0106] Sub-step 1: determine the above word-unit subsequence as the target word-unit subsequence.

[0107] Sub-step 2: Based on the target word-gram subsequence and each of the word-gram subsequences, determine the word-gram quantity information corresponding to the target word-gram subsequence. The word-gram quantity information may be the number of each word-gram subsequence in each of the word-gram subsequences that is identical to the target word-gram subsequence. In practice, the execution subject may generate the number of each word-gram subsequence in each of the word-gram subsequences that is identical to the target word-gram subsequence through a string function as the word-gram quantity information. The string function may be a function that can generate the number of times a substring appears in a specified string. For example, the string function may be str.count().

[0108] Sub-step three: Based on the target word-gram subsequence and each of the sample word-gram subsequences, determine sample quantity information corresponding to the target word-gram subsequence. The sample quantity information may be the number of sample word-gram subsequences in each of the sample word-gram subsequences that have the same content as the target word-gram subsequence. In practice, the execution entity may generate the number of sample word-gram subsequences in each of the sample word-gram subsequences that are identical to the target word-gram subsequence using the string function as the sample quantity information.

[0109] Sub-step 4: Determine the target quantity information corresponding to the target word subsequence based on the word quantity information and the sample quantity information. The target quantity information may be the word quantity information or the sample quantity information. In practice, in response to determining that the word quantity information is greater than or equal to the sample quantity information, the sample quantity information may be determined as the target quantity information. In response to determining that the word quantity information is less than the sample quantity information, the word quantity information may be determined as the target quantity information.

[0110] In the fourth sub-step, the sum of the determined target quantity information is determined as the summed quantity information.

[0111] In the fifth sub-step, the quantity corresponding to each of the above-mentioned word-unit sub-sequences is determined as the total quantity information.

[0112] In a sixth sub-step, a ratio of the sum quantity information to the total quantity information is determined as word unit precision data corresponding to the word unit quantity.

[0113] In the seventh sub-step, the quantity corresponding to each of the sample word-unit sub-sequences is determined as the total number of samples.

[0114] In an eighth sub-step, a ratio of the summed quantity information to the total sample quantity information is determined as word-unit recall data corresponding to the word-unit quantity.

[0115] In the fourth step, the number of word-grams that meets a preset number condition among the above-mentioned word-gram numbers is determined as the target word-gram number. The preset number condition may be that the word-gram number has the largest value among the above-mentioned word-gram numbers.

[0116] In the fifth step, the inverse of the target word number is determined as word weight data.

[0117] In the sixth step, based on each word-gram precision data in the determined word-gram precision data, the logarithm of the word-gram precision data is determined as word-gram logarithm data.

[0118] In the seventh step, based on each word-gram logarithmic data in the determined word-gram logarithmic data, a product of the word-gram logarithmic data and the word-gram weight data is determined as word-gram product data.

[0119] In the eighth step, the sum of the determined word unit product data is determined as the word unit sum data.

[0120] In the ninth step, based on the preset exponential base data and the above-mentioned word-unit sum data, word-unit index data is generated. The above-mentioned exponential base data may be a natural constant e. The above-mentioned word-unit index data may be a numerical value generated by the above-mentioned exponential base data and the above-mentioned word-unit sum data. In practice, the above-mentioned execution entity may determine the word-unit index data as the power of the above-mentioned word-unit sum data of the above-mentioned exponential base data. For example, when the above-mentioned word-unit sum data is 3, the corresponding word-unit index data is e raised to the power of 3.

[0121] In the tenth step, based on the above-mentioned behavior description information, the above-mentioned sample behavior description information and the preset data, the word-gram difference data is generated. The above-mentioned word-gram difference data may be a numerical value generated by the above-mentioned behavior description information, the above-mentioned sample behavior description information and the above-mentioned preset data. The above-mentioned preset data may be 1. In practice, first, the above-mentioned execution entity may determine the number of each word gram included in the above-mentioned behavior description information as word-gram quantity data. Secondly, the number of each sample word gram included in the above-mentioned sample behavior description information may be determined as sample word-gram quantity data. Then, in response to determining that the above-mentioned word-gram quantity data is greater than or equal to the above-mentioned sample word-gram quantity data, the above-mentioned sample word-gram quantity data may be determined as target word-gram quantity data. In response to determining that the above-mentioned word-gram quantity data is less than the above-mentioned sample word-gram quantity data, the above-mentioned word-gram quantity data may be determined as target word-gram quantity data. Then, the ratio of the determined target word-gram quantity data to the above-mentioned word-gram quantity data may be determined as word-gram ratio data. Finally, the difference between the above-mentioned preset data and the above-mentioned word-gram ratio data may be determined as word-gram difference data.

[0122] In the eleventh step, weight factor data is generated based on the exponential base data and the word-unit difference data. The weight factor data may be a numerical value generated from the exponential base data and the word-unit difference data. In practice, the execution entity may determine the weight factor data as the power of the word-unit difference data of the exponential base data. For example, when the exponential base data is e and the word-unit difference data is 4, the corresponding weight factor data is e raised to the power of 4.

[0123] In the twelfth step, the product of the weight factor data and the word-gram index data is determined as the precision index data.

[0124] Step 13: Generate recall index data based on the determined word-gram recall data. The recall index data may be a weighted average value corresponding to the above-mentioned word-gram recall data. In practice, for each word-gram recall data in the above-mentioned word-gram recall data, first, the execution entity may determine the inverse of the number of word-grams corresponding to the above-mentioned word-gram recall data as word-gram inverse data. Then, the product of the above-mentioned word-gram inverse data and the above-mentioned word-gram recall data may be determined as recall data. Finally, the sum of the obtained recall data may be determined as recall index data.

[0125] Step 14: Generate text accuracy data based on the behavior description information and the sample behavior description information. The text accuracy data may be a numerical value representing the accuracy of the behavior description information compared to the sample behavior description information. In practice, for each word in the behavior description information, the execution entity may use the string function to retrieve the word from the sample behavior description information and obtain a return value output by the string function. The return value may be a null value or a count of occurrences. The count of occurrences may be the number of times the word appears in the sample behavior description information. Then, among the obtained return values, each return value that meets a preset return condition may be determined as a target return value. The preset return condition may be that the return value is not null. Then, the number corresponding to each target return value may be determined as return value quantity data. Then, the number of each word in the behavior description information may be determined as pending quantity data. Finally, the ratio of the return value quantity data to the pending quantity data may be determined as text accuracy data.

[0126] Step 15: Generate text index data based on the precision index data, the recall index data, and the text precision data. In practice, the execution entity may determine the text index data as the average of the precision index data, the recall index data, and the text precision data.

[0127] The above technical solution and its related contents, combined with steps 101 to 109, serve as an inventive feature of an embodiment of the present disclosure, addressing the issue of "wasted computing resources." Factors that often lead to high computing resource consumption include: When performing behavior recognition on a video using feature extraction algorithms and classifiers, interference from background information in the video can easily occur, resulting in low recognition accuracy and, in turn, the need to re-consume computing resources to re-recognize previously recognized videos. Addressing these factors can reduce computing resource waste. To achieve this, the present disclosure first identifies each word element included in the behavior description information as a word element sequence. Second, it identifies each sample word element included in the sample behavior description information as a sample word element sequence. This generates the data to be processed. Then, for each of a predetermined number of word elements, the following steps are performed: First, based on the number of word elements and the word element sequence, each word element subsequence is generated. This generates each word element subsequence corresponding to the number of word elements. Second, based on the number of word elements and the sample word element sequence, each sample word element subsequence is generated. Thus, each sample word-gram subsequence corresponding to the aforementioned word-gram quantity can be obtained. Then, for each word-gram subsequence in each of the above word-gram subsequences, the following steps are performed: First, the above word-gram subsequence is determined as a target word-gram subsequence. Thus, a target word-gram subsequence can be obtained. Second, based on the above target word-gram subsequence and each of the above word-gram subsequences, word-gram quantity information corresponding to the above target word-gram subsequence is determined. Thus, the number of word-gram subsequences identical to the target word-gram subsequence in each word-gram subsequence can be obtained. Then, based on the above target word-gram subsequence and each of the above sample word-gram subsequences, sample quantity information corresponding to the above target word-gram subsequence is determined. Thus, the number of sample word-gram subsequences identical to the target word-gram subsequence in each sample word-gram subsequence can be obtained. Then, based on the above word-gram quantity information and the above sample quantity information, target quantity information corresponding to the above target word-gram subsequence is determined. Then, the sum of the determined target quantity information is determined as summed quantity information. Thus, summed quantity information can be obtained. Then, the quantity corresponding to each of the above word-gram subsequences is determined as summed quantity information. Thus, the total quantity information can be obtained. Then, the ratio of the above-mentioned summed quantity information to the above-mentioned summed quantity information is determined as the word-unit precision data corresponding to the above-mentioned word-unit quantity. Thus, the accuracy of the generated behavior description information can be obtained. Then, the quantity corresponding to each of the above-mentioned sample word-unit subsequences is determined as the sample total quantity information. Thus, the sample total quantity information can be obtained. Then, the ratio of the above-mentioned summed quantity information to the above-mentioned sample total quantity information is determined as the word-unit recall data corresponding to the above-mentioned word-unit quantity. Thus, the recall rate of the generated behavior description information can be obtained. Then, the number of word units that meet the preset quantity conditions among the above-mentioned each word unit quantity is determined as the target word unit quantity.Then, the inverse of the target number of word units is determined as word unit weight data. Then, based on each word unit precision data in the determined word unit precision data, the logarithm of the word unit precision data is determined as word unit logarithm data. Then, based on each word unit logarithm data in the determined word unit logarithm data, the product of the word unit logarithm data and the word unit weight data is determined as word unit product data. Then, the sum of the determined word unit product data is determined as word unit sum data. Thus, word unit sum data can be obtained. Then, based on the preset exponential base data and the word unit sum data, word unit index data is generated. Thus, word unit index data can be obtained. Then, based on the behavior description information, the sample behavior description information, and the preset data, word unit difference data is generated. Then, based on the exponential base data and the word unit difference data, weight factor data is generated. Thus, weight factor data can be obtained. Then, the product of the weight factor data and the word unit index data is determined as precision index data. Thus, precision index data can be obtained. Then, based on the determined recall data of each word unit, recall index data is generated. Thus, the weighted average value of the recall data of each word unit can be obtained. Then, based on the above-mentioned behavior description information and the above-mentioned sample behavior description information, text precision data is generated. Thus, the accuracy of the generated behavior description information can be obtained. Finally, based on the above-mentioned precision index data, the above-mentioned recall index data and the above-mentioned text precision data, text index data is generated. Thus, the corresponding quality index of the generated behavior description information can be obtained. Also, because the network parameters of the initial neural network can be adjusted according to the quality index of the generated behavior description information, the accuracy of the behavior description information generated by the trained behavior recognition model can be improved through continuous iteration, thereby reducing the probability of consuming computing resources to perform behavior recognition on the recognized video again due to low accuracy, thereby reducing the waste of computing resources.

[0128] Step 107: input the masked question text data into the text processing layer in the behavior recognition model to obtain video text feature information.

[0129] In some embodiments, the execution entity may input the masked question text data into the text processing layer of the behavior recognition model to obtain video text feature information. The text processing layer includes a text mapping network, a text embedding layer, and a text linear layer.

[0130] Step 108: Input the video frame feature information and the video text feature information into the output layer of the behavior recognition model to obtain behavior description information.

[0131] In some embodiments, the execution entity may input the video frame feature information and the video text feature information into the output layer of the behavior recognition model to obtain behavior description information.

[0132] Step 109: Send the behavior description information to the user terminal.

[0133] In some embodiments, the execution entity may send the behavior description information to the user terminal.

[0134] The above-described embodiments of the present disclosure have the following beneficial effects: The large-scale video action recognition method based on Fourier transform saliency in some embodiments of the present disclosure can reduce the waste of computing resources. Specifically, the reason for this waste of computing resources is that when identifying actions in a video, the recognition process is easily interfered with by background information in the video, resulting in low recognition accuracy. This in turn leads to a high probability of requiring computing resources to re-identify the video, which in turn leads to a waste of computing resources. Furthermore, when using a feature extraction algorithm to extract features from a video, each frame in the video must be processed, resulting in a high consumption of computing resources. Based on this, the large-scale video action recognition method based on Fourier transform saliency in some embodiments of the present disclosure first generates a set of to-be-processed video frames corresponding to the to-be-processed video data in response to receiving to-be-processed video data and question text data sent by a user terminal. This generates the video data to be identified. Secondly, based on the to-be-processed video frame set, frequency domain transform matrices corresponding to the to-be-processed video frame set are generated. Thus, by performing Fourier transform processing on the video, the model can better identify foreground information during recognition and reduce interference from background information. Then, based on each of the frequency domain transformation matrices, a target video frame corresponding to the video data to be processed is determined. This allows the most significant video frame to be processed to be determined from the group of video frames to be processed. Then, based on the question text data, target question text data is generated. Then, the target question text data is masked to obtain masked question text data. This allows redundant information in the text data to be masked. Then, the target video frame is input into the image processing layer of a pre-trained behavior recognition model to obtain video frame feature information. The image processing layer includes an image mapping network, an image embedding network, and an image linearization layer. Thus, the target video frame can be processed using the behavior recognition model. Then, the masked question text data is input into the text processing layer of the behavior recognition model to obtain video text feature information. The text processing layer includes a text mapping network, a text embedding layer, and a text linearization layer. Thus, the masked question text data can be processed using the behavior recognition model. Then, the video frame feature information and the video text feature information are input into the output layer of the behavior recognition model to obtain behavior description information. Thus, behavior description information corresponding to the video can be obtained. Finally, the behavior description information is sent to the user terminal. Furthermore, because the most significant video frame to be processed can be identified from the group of video frames to be processed, and then processed to obtain the behavior description information, it is not necessary to process every frame in the video data to be processed. This reduces the computing resources consumed during processing.In addition, because the video can be processed through Fourier transform first, the model can better process foreground information and reduce the interference of background information on recognition, thereby improving the recognition accuracy and reducing the probability of needing to consume computing resources to re-recognize the video, thereby reducing the waste of computing resources.

[0135] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a large-scale video behavior recognition device based on Fourier transform saliency. These device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0136] like Figure 2 As shown, some embodiments of the large-scale video behavior recognition device 200 based on Fourier transform saliency include: a first generating unit 201, a second generating unit 202, a determining unit 203, a third generating unit 204, a masking unit 205, a first input unit 206, a second input unit 207, a third input unit 208 and a sending unit 209. The first generating unit 201 is configured to generate a group of to-be-processed video frames corresponding to the to-be-processed video data in response to receiving the to-be-processed video data and question text data sent by the user terminal; the second generating unit 202 is configured to generate each frequency domain transformation matrix corresponding to the to-be-processed video frame group based on the to-be-processed video frame group; the determining unit 203 is configured to determine the target video frame corresponding to the to-be-processed video data based on each frequency domain transformation matrix; the third generating unit 204 is configured to generate target question text data based on the question text data; the masking unit 205 is configured to perform masking on the target question text data to obtain masked question text data; the first input unit 206 is configured The target video frame is input into the image processing layer of the pre-trained behavior recognition model to obtain video frame feature information, wherein the image processing layer includes an image mapping network, an image embedding network and an image linear layer; the second input unit 207 is configured to input the masked question text data into the text processing layer of the behavior recognition model to obtain video text feature information, wherein the text processing layer includes a text mapping network, a text embedding layer and a text linear layer; the third input unit 208 is configured to input the video frame feature information and the video text feature information into the output layer of the behavior recognition model to obtain behavior description information; the sending unit 209 is configured to send the behavior description information to the user terminal.

[0137] It is understandable that the various units recorded in the large-scale video behavior recognition device 200 based on Fourier transform saliency are similar to those in the reference Figure 1Therefore, the operations, features and beneficial effects described above for the method are also applicable to the large-scale video behavior recognition device 200 based on Fourier transform saliency and the units included therein, and will not be repeated here.

[0138] Reference below Figure 3 , which shows a structural diagram of an electronic device (such as a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0139] like Figure 3 As shown, electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 302 or programs loaded from a storage device 308 into a random access memory (RAM) 303. RAM 303 also stores various programs and data required for the operation of electronic device 300. Processing device 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0140] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0141] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0142] It should be noted that the computer-readable medium described in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. Furthermore, in some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0143] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0144] The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist independently without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: generates a group of video frames to be processed corresponding to the above-mentioned video data to be processed in response to receiving the video data to be processed and the question text data sent by the user terminal; generates each frequency domain transformation matrix corresponding to the above-mentioned video frame group to be processed based on the above-mentioned video frame group; determines the target video frame corresponding to the above-mentioned video data to be processed based on the above-mentioned frequency domain transformation matrix; generates target question text data based on the above-mentioned question text data; performs mask processing on the above-mentioned target question text data to obtain a masked question. text data; inputting the target video frame into the image processing layer of the pre-trained behavior recognition model to obtain video frame feature information, wherein the image processing layer includes an image mapping network, an image embedding network and an image linear layer; inputting the masked problem text data into the text processing layer of the behavior recognition model to obtain video text feature information, wherein the text processing layer includes a text mapping network, a text embedding layer and a text linear layer; inputting the video frame feature information and the video text feature information into the output layer of the behavior recognition model to obtain behavior description information; and sending the behavior description information to the user terminal.

[0145] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor. For example, they may be described as follows: a processor comprising a first generation unit, a second generation unit, a determination unit, a third generation unit, a mask unit, a first input unit, a second input unit, a third input unit, and a sending unit. The names of these units do not, in some cases, constitute limitations on the units themselves. For example, the sending unit may also be described as a "unit that sends the above-mentioned behavior description information to the above-mentioned user terminal."

[0148] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0149] The above descriptions are merely some preferred embodiments of the present disclosure and illustrate the underlying technical principles. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A large-scale video action recognition method based on Fourier transform saliency, comprising: In response to receiving the to-be-processed video data and question text data sent by the user terminal, generating a to-be-processed video frame group corresponding to the to-be-processed video data; Generating frequency domain transformation matrices corresponding to the video frame group to be processed based on the video frame group to be processed, wherein each video frame to be processed in the video frame group to be processed corresponds to image height data and image width data; and generating frequency domain transformation matrices corresponding to the video frame group to be processed based on the video frame group to be processed, comprising: For each to-be-processed video frame in the to-be-processed video frame group, perform the following steps: Based on a preset first gamma coefficient value, a preset second gamma coefficient value, and a preset third gamma coefficient value, performing image conversion processing on the video frame to be processed to obtain grayscale video frame data; Performing frequency domain transformation processing on the grayscale video frame data to obtain a frequency domain transformation matrix; Determining a target video frame corresponding to the video data to be processed based on the respective frequency domain transformation matrices, wherein determining the target video frame corresponding to the video data to be processed based on the respective frequency domain transformation matrices includes: For each of the frequency domain transformation matrices, the following steps are performed: generating a spectrum energy matrix based on the frequency domain transformation matrix; generating spectrum element accumulation data based on the spectrum energy matrix; Determine the product of the image height data and the image width data corresponding to the video frame to be processed as image processing data; determining a ratio of the spectrum element accumulated data to the image processed data as average energy data; Determining the average energy data that meets the preset energy data condition among the determined average energy data as the target average energy data; Determining the to-be-processed video frame corresponding to the target average energy data in the to-be-processed video frame group as a target video frame; Based on the question text data, generating target question text data; Performing masking on the target question text data to obtain masked question text data; Inputting the target video frame into an image processing layer in a pre-trained behavior recognition model to obtain video frame feature information, wherein the image processing layer includes an image mapping network, an image embedding network, and an image linear layer; Inputting the masked question text data into a text processing layer in the behavior recognition model to obtain video text feature information, wherein the text processing layer includes a text mapping network, a text embedding layer, and a text linear layer; Inputting the video frame feature information and the video text feature information into the output layer of the behavior recognition model to obtain behavior description information; The behavior description information is sent to the user terminal.

2. The method according to claim 1, wherein The performing frequency domain transformation processing on the grayscale video frame data to obtain a frequency domain transformation matrix includes: Determining each matrix height data and each matrix width data based on the image height data and image width data corresponding to the video frame to be processed; Determining a grayscale video matrix based on the grayscale video frame data; For each matrix height data in the respective matrix height data and each matrix width data in the respective matrix width data, the following steps are performed: For each element of the grayscale video matrix, perform the following steps: Determining element height data and element width data corresponding to the element based on the grayscale video matrix and the element; Determine the product of the matrix height data and the element height data as matrix element height data; Determine the product of the matrix width data and the element width data as matrix element width data; Determine the ratio of the matrix element height data to the image height data corresponding to the video frame to be processed as height ratio data; Determine a ratio of the matrix element width data to the image width data corresponding to the to-be-processed video frame as width ratio data; determining a sum of the height ratio data and the width ratio data as sum ratio data; Determining the product of the preset imaginary number data, the preset coefficient data and the sum ratio data as exponential data; Generate index data to be processed based on preset index base data and the index data; Determine the product of the element value corresponding to the element and the index data to be processed as matrix sub-element data; Determining the sum of the determined matrix sub-element data as matrix element data corresponding to the matrix height data and the matrix width data; The obtained matrix element data are combined into a frequency domain transformation matrix.

3. The method according to claim 1, wherein The behavior recognition model is trained by the following steps: Acquire a sample set, wherein each sample in the sample set includes a sample video frame, sample mask question text data, and sample behavior description information, wherein the sample behavior description information includes each sample word; Based on the sample set, the following training steps are performed: Inputting each sample video frame included in at least one sample in the sample set into an image mapping network in an image processing layer of an initial neural network to obtain each sample video frame information, wherein each sample video frame information in the each sample video frame information corresponds to one sample; Inputting each sample video frame information corresponding to the at least one sample into an image embedding network in an image processing layer of an initial neural network to obtain each sample weighting matrix, wherein each sample weighting matrix in the each sample weighting matrix corresponds to one sample; Inputting each sample weighting matrix corresponding to the at least one sample into the image linear layer of the image processing layer of the initial neural network to obtain feature information of each sample video frame corresponding to the at least one sample; Inputting each sample masked question text data included in the at least one sample into a text mapping network in a text processing layer of an initial neural network to obtain feature information of each sample question text, wherein each sample question text feature information in the each sample question text feature information corresponds to one sample; Inputting each sample question text feature information corresponding to the at least one sample into a text embedding layer in a text processing layer of an initial neural network to obtain each sample text embedding feature information, wherein each sample text embedding feature information in the each sample text embedding feature information corresponds to one sample; Inputting each sample text embedding feature information corresponding to the at least one sample into a text linear layer in a text processing layer of an initial neural network to obtain each sample video text feature information, wherein each sample video text feature information in the each sample video text feature information corresponds to one sample; For each sample of the at least one sample, generating alignment loss data corresponding to the sample based on the feature information of the video frame of each sample and the feature information of the video text of each sample; In response to determining that the generated alignment loss data do not meet the preset loss conditions, the network parameters of the image processing layer and the text processing layer in the initial neural network are adjusted, and a sample set is composed using unused samples, and the image processing layer and the text processing layer of the adjusted initial neural network are used to perform the training step again.

4. The method according to claim 3, wherein: After generating alignment loss data corresponding to each sample of the at least one sample based on the respective sample video frame feature information and the respective sample video text feature information, the method further includes: In response to determining that each of the generated alignment loss data satisfies a preset loss condition, the following steps are performed: Inputting each sample video frame feature information and each sample video text feature information corresponding to the at least one sample into the output layer of the initial neural network to obtain each behavior description information corresponding to the at least one sample, wherein each behavior description information includes each word; For each of the at least one sample, generating text indicator data corresponding to the sample based on the behavior description information corresponding to the sample in the respective behavior description information and the sample behavior description information included in the sample; In response to determining that each of the generated text indicator data satisfies a preset indicator data condition, determining the initial neural network as a behavior recognition model; In response to determining that the generated text indicator data do not meet the indicator data conditions, the network parameters of the output layer in the initial neural network are adjusted, and a sample set is composed using unused samples, and the output layer of the adjusted initial neural network is used to perform the training step again.

5. A large-scale video behavior recognition device based on Fourier transform saliency, comprising: A first generating unit is configured to generate a to-be-processed video frame group corresponding to the to-be-processed video data in response to receiving the to-be-processed video data and question text data sent by the user terminal; The second generating unit is configured to generate, based on the to-be-processed video frame group, frequency domain transformation matrices corresponding to the to-be-processed video frame group, wherein each to-be-processed video frame in the to-be-processed video frame group corresponds to image height data and image width data; and the generating, based on the to-be-processed video frame group, frequency domain transformation matrices corresponding to the to-be-processed video frame group comprises: For each to-be-processed video frame in the to-be-processed video frame group, perform the following steps: Based on a preset first gamma coefficient value, a preset second gamma coefficient value, and a preset third gamma coefficient value, performing image conversion processing on the video frame to be processed to obtain grayscale video frame data; Performing frequency domain transformation processing on the grayscale video frame data to obtain a frequency domain transformation matrix; The determining unit is configured to determine a target video frame corresponding to the video data to be processed based on the respective frequency domain transformation matrices, wherein determining the target video frame corresponding to the video data to be processed based on the respective frequency domain transformation matrices includes: For each of the frequency domain transformation matrices, the following steps are performed: generating a spectrum energy matrix based on the frequency domain transformation matrix; generating spectrum element accumulation data based on the spectrum energy matrix; Determine the product of the image height data and the image width data corresponding to the video frame to be processed as image processing data; determining a ratio of the spectrum element accumulated data to the image processed data as average energy data; Determining the average energy data that meets the preset energy data condition among the determined average energy data as the target average energy data; Determining the to-be-processed video frame corresponding to the target average energy data in the to-be-processed video frame group as a target video frame; A third generating unit is configured to generate target question text data based on the question text data; a masking unit configured to perform masking processing on the target question text data to obtain masked question text data; A first input unit is configured to input the target video frame into an image processing layer in a pre-trained behavior recognition model to obtain video frame feature information, wherein the image processing layer includes an image mapping network, an image embedding network, and an image linear layer; A second input unit is configured to input the masked question text data into a text processing layer in the behavior recognition model to obtain video text feature information, wherein the text processing layer includes a text mapping network, a text embedding layer, and a text linear layer; A third input unit is configured to input the video frame feature information and the video text feature information into an output layer of the behavior recognition model to obtain behavior description information; The sending unit is configured to send the behavior description information to the user terminal.

6. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images