Video classification systems and methods and neural network training systems and methods

Abstract

The present disclosure provides a video classification system and method, and a neural network training system and method. The video classification system collects a plurality of sample images, encodes a first image and a second image of all sample images into a feature matrix of the sample images using a convolutional neural network module, and obtains a classification of a video based on a recurrent neural network module and the feature matrix. The neural network training system collects a plurality of first sample images, trains a convolutional neural network module and a classification module using a first training sample of all first sample images to obtain a plurality of parameter values of the convolutional neural network module, and trains a recurrent neural network module based on the parameter values and a plurality of second sample images.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of video classification. In particular, techniques of applying neural networks to video classification. BACKGROUND

[0002] Animation is an important part of the television industry. They are the most popular entertainment for children, but not only that, there are more and more animated videos and movies provided for viewers of all ages. Now, not only traditional cable television, we can also watch a large number of animated videos on streaming media services. In this case, how to improve the user's experience of watching animated videos on television becomes an important problem. In order to achieve this goal, an effective system and method are needed to detect videos in real time, classify whether the video is an animation, and apply different enhancement effects to the video according to the detection result. SUMMARY

[0003] Therefore, some embodiments of the present disclosure provide a video classification system and method and a neural network training system and method, which relate to a video classification system, a video classification method, a neural network training system, and a neural network training method to improve the problems of the prior art.

[0004] An embodiment of the present disclosure provides a video classification system. The video classification system includes a processor, a convolutional neural network module, and a recurrent neural network module. The processor is configured to obtain a video. The convolutional neural network module has a plurality of trained first parameters. The recurrent neural network module has a plurality of trained second parameters. The processor is configured to perform the following steps: selecting a current time point according to a time interval, sampling the video according to the current time point to obtain a sampled image of the current time point. The sampled image of the current time point is resized to obtain a corresponding first image. The first image has a first pixel size, and the first pixel size of the first image is smaller than the pixel size of the sampled image. The sampled image of the current time point is cropped to obtain at least one partial image, and based on the at least one partial image, a second image is obtained, wherein the second image has the first pixel size. The first image and the second image of the sampled image of the current time point are encoded into a feature vector corresponding to the sampled image of the current time point using the convolutional neural network module. The feature vector corresponding to the current time point and a plurality of past feature vectors corresponding to a plurality of past time points are sequentially merged into a feature matrix. And based on the recurrent neural network module and the feature matrix, a classification of the video is obtained.

[0005] An embodiment of the present disclosure provides a video classification method performed by a processor. The video classification method comprises the following steps: selecting a current time point according to a time interval, sampling a video according to the current time point to obtain a sampled image of the current time point. The sampled image of the current time point is resized to obtain a corresponding first image. The first image has a first pixel size, and the first pixel size of the first image is smaller than the pixel size of the sampled image. The sampled image of the current time point is cropped to obtain at least one partial image, and a corresponding second image is obtained based on the at least one partial image, wherein the second image has the first pixel size. The first image and the second image of the sampled image of the current time point are encoded into a feature vector corresponding to the sampled image of the current time point using a convolutional neural network module. The feature vector corresponding to the current time point and a plurality of past feature vectors corresponding to a plurality of past time points are sequentially merged into a feature matrix.

[0006] An embodiment of the present disclosure provides a neural network training system. The neural network training system comprises a processor, a convolutional neural network module, a recurrent neural network module, and a classification module. The processor is configured to obtain a plurality of videos, a plurality of training images, and a classification corresponding to each video and corresponding to each training image. The convolutional neural network module has a plurality of first parameters, the recurrent neural network module has a plurality of second parameters, and the classification module has a plurality of third parameters. The processor is configured to perform the following steps: obtaining a plurality of first sampled images from the plurality of videos and the plurality of training images. Selecting an unselected one of the plurality of first sampled images as a current image. Resizing the current image to obtain a corresponding first image, wherein the first image has a first pixel size smaller than the pixel size of the current image. Cropping the current image to obtain at least one first partial image, and obtaining a corresponding second image based on the at least one first partial image, wherein the second image has the first pixel size. Setting the classification corresponding to the first image, the second image, and the current image as a first training sample corresponding to the current image. Repeating the above steps until all the plurality of first sampled images are selected. Training a first synthesized network composed of the convolutional neural network module and the classification module using all the first training samples corresponding to each first sampled image to obtain a plurality of first parameter values corresponding to the plurality of first parameters. Training the recurrent neural network module based on the plurality of first parameter values and a plurality of second sampled images obtained from each video to obtain a plurality of second parameter values corresponding to the plurality of second parameters.

[0007] An embodiment of the present disclosure provides a neural network training method performed by a processor. The neural network training method comprises the following steps: obtaining a plurality of first sample images from a plurality of videos and a plurality of training images. Selecting an unselected one of the plurality of first sample images as a current image. Performing pixel size adjustment on the current image to obtain a corresponding first image, wherein a first pixel size of the first image is smaller than a pixel size of the current image. Performing image cropping on the current image to obtain at least one first partial image, and obtaining a corresponding second image based on the at least one first partial image, wherein a pixel size of the second image is the first pixel size. Setting a classification corresponding to the first image, the second image and the current image as a first training sample corresponding to the current image. Repeating the above steps until all the plurality of first sample images are selected. Training a first synthetic network composed of a convolutional neural network module and a classification module using all the first training samples corresponding to each first sample image to obtain a plurality of first parameter values corresponding to a plurality of first parameters. Training a recurrent neural network module based on the plurality of first parameter values and a plurality of second sample images obtained from each video to obtain a plurality of second parameter values corresponding to a plurality of second parameters.

[0008] Based on the above, some embodiments of the present disclosure provide a video classification system, a video classification method, a neural network training system and a neural network training method. By training a convolutional neural network module and a recurrent neural network module in advance with images including two different types of information in video capture images, the video classification system can quickly obtain the classification result of a video by classifying a plurality of sample pictures of the video. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 A block diagram of a video classification system according to an embodiment of the present disclosure.

[0010] Figure 2-1 An operation schematic diagram of a video classification system according to some embodiments of the present disclosure.

[0011] Figure 2-2 An operation schematic diagram of a video classification system according to some embodiments of the present disclosure.

[0012] Figure 3 A block diagram of a neural network training system according to an embodiment of the present disclosure.

[0013] Figure 4 An operation schematic diagram of a neural network training system according to some embodiments of the present disclosure.

[0014] Figure 5 An operation schematic diagram of a neural network training system according to some embodiments of the present disclosure.

[0015] Figure 6 Structure diagram of an electronic device according to some embodiments of the present disclosure.

[0016] Figure 7 Architecture diagram of a classification module according to some embodiments of the present disclosure.

[0017] Figure 8 Flowchart of a video classification method according to some embodiments of the present disclosure.

[0018] Figure 9 Flowchart of a video classification method according to some embodiments of the present disclosure.

[0019] Figure 10 Flowchart of a video classification method according to some embodiments of the present disclosure.

[0020] Figure 11 Flowchart of a video classification method according to some embodiments of the present disclosure.

[0021] Figure 12 Flowchart of a neural network training method according to some embodiments of the present disclosure.

[0022] Figure 13 Flowchart of a neural network training method according to some embodiments of the present disclosure.

[0023] List of reference signs: 100: video classification system; 101, 301, 701: processor; 102, 302: convolutional neural network module; 103: trained first parameter; 104, 304: recurrent neural network module; 105: trained second parameter; 106, 308: video; 107: present time point; 108, 109, 110, 111: past time point; 112: time point; 201: sampled image; 202, 402: first image; 203, 403: cropped image; 205a: first partial feature vector; 205b: second partial feature vector; 205: feature vector; 207, 208, 209, 210: past feature vector; 206: feature matrix; 300: neural network training system; 303: first parameter; 305: second parameter; 306: classification module; 307: third parameter; 309: training image; 401: first sampled image; 404: first synthetic network; t0, t1, t2, t3, t4, t5, t6…t n: time point; 500, 501, 502, 503, 504, 505, 506,..., 50n: second sampling image; 60, 61, 62: sample feature matrix; 600-606: vector; 700: electronic device; 702: internal memory; 703: non-volatile memory; 801: input layer; 802: hidden layer; 803: output layer; 804: normalized exponential function module; T: time interval; y1, y2: output; x1, x2,..., x 1024 : input signal; S901-S906, S1001-S1002, S1101-S1102, S1201-S1202, S1301-S1308, S1401-S1411: step. DETAILED DESCRIPTION

[0024] The foregoing and other technical contents, features and effects of the present disclosure will be more clearly understood from the following detailed description of embodiments with reference to the accompanying drawings. The thickness or size of each element in the drawings is exaggerated or omitted or schematized for the purpose of understanding and reading by those of ordinary skill in the art, and the size of each element is not completely the actual size, and is not intended to limit the scope of the present disclosure, and therefore does not have technical significance. Any modification, change in proportion or adjustment in size, without affecting the effects and purposes that can be achieved by the present disclosure, shall still fall within the scope of the technical content disclosed by the present disclosure. The same reference numerals will be used to represent the same or similar elements in all the drawings.

[0025] Figure 1 A block diagram of a video classification system according to an embodiment of the present disclosure is shown. Referring to Figure 1 , the video classification system 100 includes a processor 101, a convolutional neural network module 102, and a recurrent neural network module 104. The processor 101 is configured to obtain a video 106. The convolutional neural network module 102 has a plurality of trained first parameters 103. The recurrent neural network module 104 has a plurality of trained second parameters 105. The video 106 can be classified into two categories, animation and non-animation.

[0026] In some embodiments of the present disclosure, the video classification system 100 is implemented on a single-chip microcontroller (e.g., 8051), the processor 101 is a central processing unit (CPU) of the single-chip microcontroller, the convolutional neural network module 102 is a model with a convolutional neural network (CNN) structure stored in a memory of the single-chip microcontroller (e.g., a model with a convolutional neural network structure designed in Pytorch or Tensorflow), and the recurrent neural network module 104 is a model with a recurrent neural network (RNN) structure stored in a memory of the single-chip microcontroller (e.g., a model with a recurrent neural network structure designed in Pytorch or Tensorflow). The trained first parameters 103 and the trained second parameters 105 are memory locations storing values of trained parameters.

[0027] In some embodiments of the present disclosure, the video classification system 100 is implemented on a network system, the convolutional neural network module 102 is a model with a convolutional neural network structure stored in a network space, and the recurrent neural network module 104 is a model with a recurrent neural network structure stored in the network space. The processor 101 accesses the convolutional neural network module 102 and the recurrent neural network module 104 through the network.

[0028] The following describes in detail the video classification method and the cooperation between the modules of the video classification system 100 according to some embodiments of the present disclosure.

[0029] Figure 2-1 、 Figure 2-2 The video classification system 100 according to some embodiments of the present disclosure is shown in the operation schematic diagram. Figure 8 The video classification method according to some embodiments of the present disclosure is shown in the flowchart, which is described in combination with Figure 1 、 Figure 2-1 、 Figure 2-2 and Figure 8 In step S901, the processor 101 selects the current time point 107 according to a time interval T. The processor 101 samples the video according to the current time point 107 to obtain a sampled image 201 of the current time point 107.

[0030] In step S902, the processor 101 performs pixel size adjustment on the sampled image 201 of the current time point 107 to obtain a corresponding first image 202. The pixel size of the first image 202 is a first pixel size. The first pixel size of the first image is smaller than the pixel size of the sampled image 201. In some embodiments of the present disclosure, the processor 101 performs pixel size adjustment on the sampled image 201 of the current time point 107 to obtain the first image 202. Figure 2-2In the illustrated example, the first pixel size is 224×224×3, and the pixel size of the sampled image 201 is 3840×2160×3, where 3 represents the three components of red (R), green (G), and blue (B). It should be noted that in this embodiment, the first pixel size is chosen to be 224×224×3, and the pixel size of the sampled image is chosen to be 3840×2160×3. However, the choice of pixel size must be based primarily on the resolution of the video or player, and this disclosure is not limited thereto.

[0031] In step S903, the processor 101 performs image cropping on the sampled image at the current time point 107 to obtain at least one corresponding partial image. In this embodiment, the processor 101 performs image cropping at the middle position of the sampled image to obtain a cropped image 203 with a pixel size of a first pixel size (224×224×3 in this example). The cropped image 203 is then used as the second image. That is, in this embodiment, there is only one partial image, and the second image is equivalent to the partial image.

[0032] Figure 9 The flowchart illustrates a video classification method based on some embodiments of this disclosure. Please also refer to... Figure 9 ,exist Figure 9 In the illustrated embodiment, step S903 further includes steps S1001 and S1002. In step S1001, the processor 101 performs image cropping at other fixed positions of the sampled image 201 at the current time point 107 (different from the intermediate position in the aforementioned embodiment) to obtain a cropped image 203 with a pixel size of a first pixel size (224×224×3 in this example). In step S1002, the cropped image 203 is used as the second image.

[0033] Figure 10 The flowchart illustrates a video classification method based on some embodiments of this disclosure. Please also refer to... Figure 10 ,exist Figure 10 In the illustrated embodiment, step S903 further includes steps S1101 and S1102. In step S1101, processor 101 performs image cropping at multiple corresponding positions of the sampled image 201 at the current time point 107 to obtain multiple cropped images with a pixel size of a first pixel size (224×224×3 in this example). In step S1102, processor 101 uses these cropped images as at least one of the aforementioned partial images. Processor 101 then averages the value of each pixel to average these cropped images, resulting in a second image.

[0034] In step S904, the processor 101 encodes the first image and the second image of the sampled image 201 of the present time point 107 into a feature vector 205 corresponding to the sampled image of the present time point 107 using the convolutional neural network module 102. In step S905, the processor 101 combines the feature vector 205 of the present time point 107 and the past feature vectors 207, 208, 209, 210 obtained from the previous four past time points 108, 109, 110, 111 in sequence into a feature matrix 206. The past feature vector 207 is a past feature vector corresponding to the past time point 108, the past feature vector 208 is a past feature vector corresponding to the past time point 109, and so on. In step S906, the processor 101 obtains the classification of the video 106 based on the recurrent neural network module 104 and the feature matrix. In this embodiment, the classification of the video 106 is animation and non-animation. It should be noted that in this embodiment, the processor 101 combines the past feature vectors 207, 208, 209, 210 obtained from the previous four past time points 108, 109, 110, 111 and the feature vector 205 of the present time point 107 to obtain the feature matrix 206, but the present disclosure is not limited to combining the past feature vectors of the previous four past time points, and other numbers of past feature vectors of past time points can also be combined.

[0035] Figure 11 A flowchart of a video classification method according to some embodiments of the present disclosure is shown. Please refer to Figure 11 , and please refer to Figure 2-1 , Figure 2-2 . In the embodiment shown in Figure 11 , step S904 further includes step S1201 and step S1202. In step S1201, the processor 101 encodes the first image of the sampled image 201 into a first partial feature vector 205a using the convolutional neural network module 102, where the dimension of the first partial feature vector 205a is 512x1. The processor 101 encodes the second image of the sampled image 201 into a second partial feature vector 205b using the convolutional neural network module 102, where the dimension of the second partial feature vector 205b is 512x1. In step S1202, the processor 101 combines the first partial feature vector 205a and the second partial feature vector 205b to obtain the feature vector 205, where the dimension of the feature vector 205 is 1024x1.

[0036] In some embodiments of the present disclosure, after the current time point 107, the processor 101 selects a time point 112 as the current time point again according to the time interval T, and repeats the above processing procedure. When the processor 101 sets the time point 112 as the current time point, the original current time point 107 is set by the processor 101 as a past time point relative to the time point 112, and the feature vector 205 corresponding to the current time point 107 is set by the processor 101 as a past feature vector.

[0037] In some embodiments of the present disclosure, after the processor 101 obtains a sufficient number of past feature vectors of past time points (four in the foregoing embodiment), the processor 101 obtains the classification of the video 106 based on the recurrent neural network module 104 and the feature matrix.

[0038] In some embodiments of the present disclosure, the convolutional neural network module 102 is a modified ShuffleNet V2 model. The modified ShuffleNet V2 model is a modified output layer of the ShuffleNet V2, so that the modified ShuffleNet V2 model can input an input image with a pixel size of 224x224x3 and output a vector with a dimension of 512x1. This 512x1 vector is the feature vector corresponding to the input image generated by the ShuffleNet V2 model.

[0039] In addition, it is worth noting that, due to the unique design structure, the long short-term memory (LSTM) in the recurrent neural network is suitable for processing and predicting important events with very long intervals and delays in time series. Therefore, in an embodiment of the present disclosure, the recurrent neural network module 104 is selected as the long short-term memory.

[0040] In the foregoing embodiments, how to use the convolutional neural network module 102 with trained parameters and the recurrent neural network module 104 to detect the classification (animation and non-animation) of the video 106 is mainly disclosed. In the following embodiments, how to obtain these trained parameters will be disclosed.

[0041] Figure 3 is a block diagram of a neural network training system according to an embodiment of the present disclosure. Please refer to Figure 3The neural network training system 300 includes a processor 301, a convolutional neural network module 302, a recurrent neural network module 304, and a classification module 306. The processor 301 is configured to obtain a plurality of videos 308, a plurality of training images 309, and a classification (in this embodiment, animation and non-animation) corresponding to each video 308 and each training image 309. The convolutional neural network module 302 has a plurality of first parameters 303. The recurrent neural network module 304 has a plurality of second parameters 305. The classification module 306 has a plurality of third parameters 307.

[0042] In some embodiments of the present disclosure, the neural network training system 300 is implemented on a server, the processor 101 is a central processing unit (CPU) or a tensor processing unit (TPU) of the server, the convolutional neural network module 302 is a model with a convolutional neural network structure stored in the server memory (for example, a model with a convolutional neural network structure designed in Pytorch or Tensorflow). The recurrent neural network module 304 is a model with a recurrent neural network structure stored in the server memory (for example, a model with a recurrent neural network structure designed in Pytorch or Tensorflow). The classification module 306 is a model with a multiclass classifier structure stored in the server memory (for example, a model with a multiclass classifier structure designed in Pytorch or Tensorflow). The first parameters 303, the second parameters 305, and the third parameters 307 are memory locations where parameter values are stored.

[0043] The following describes in detail the operation of the neural network training method and the cooperation between the modules of the neural network training system 300 according to some embodiments of the present disclosure.

[0044] Figure 4 The neural network training system 300 operation schematic diagram according to some embodiments of the present disclosure is shown. Figure 12 The neural network training method flowchart according to some embodiments of the present disclosure is shown. Please refer to Figure 3 、 Figure 4 and Figure 12In step S1301, the processor 301 obtains a plurality of first sample images 401 from the received video 308 and the training images 309. The classification of each of the first sample images 401 is the same as the classification of the source video 308 or the training images 309. For example, one of the plurality of first sample images 401 is obtained from an animation video, and the classification of this image is animation. For another example, one of the plurality of first sample images 401 is selected from the training images 309, and the classification of this image is the classification of the selected image from the training images 309. In step S1302, the processor 301 selects one of the plurality of first sample images 401 that is not selected as a current image.

[0045] In step S1303, the processor 301 performs pixel size adjustment on the selected current image to obtain a corresponding first image 402. The first image 402 has a first pixel size, and the first pixel size of the first image is smaller than the pixel size of the selected current image. Figure 4 In the illustrated example, the first pixel size is 224x224x3, and the pixel size of the current image is 3840x2160x3, where 3 represents three components of red (R), green (G), and blue (B). It should be noted that in this embodiment, the first pixel size is selected to be 224x224x3, and the pixel size of the current image is 3840x2160x3, but the selection of the pixel size should be mainly based on the resolution of the video, and the present disclosure is not limited thereto.

[0046] In step S1304, the processor 301 performs image cropping on the current image to obtain at least one first partial image, and obtains a corresponding second image based on the at least one first partial image. In this embodiment, the processor 301 performs image cropping at the middle position of the current image to obtain a cropped image 403 having a pixel size of the first pixel size (224x224x3 in this example). The cropped image 403 is taken as the second image. In this embodiment, the number of first partial images is one, and the second image is equivalent to the first partial image.

[0047] In some embodiments of the present disclosure, the processor 301 performs image cropping at other fixed positions to obtain a cropped image 403 having a pixel size of the first pixel size (224x224x3 in this example). The cropped image 403 is taken as the second image.

[0048] In some embodiments of the present disclosure, the processor 301 performs image cropping at the corresponding positions of the current image to obtain a plurality of cropped images with a first pixel size (224x224x3 in this example). These cropped images are the aforementioned at least one partial image. The processor 301 further averages the values of each pixel to average the cropped images as a second image.

[0049] In step S1305, the processor 301 sets the classification corresponding to the first image, the second image, and the current image as a first training sample corresponding to the current image. In step S1306, the processor 301 repeats steps S1302, S1303, S1304, and S1305 until all the first sample images are selected. After all the first sample images are selected, each first sample image has a corresponding first training sample. In step S1307, the first synthetic network 404 composed of the convolutional neural network module 302 and the classification module 306 is trained using all the first training samples corresponding to each first sample image to obtain a plurality of first parameter values corresponding to the plurality of first parameters 303.

[0050] After step S1307, the processor 301 has obtained a plurality of first parameter values corresponding to the plurality of first parameters 303, that is, the trained parameters of the convolutional neural network module 302. Next, the processor 301 obtains a plurality of second parameter values of the second parameters 305 of the recurrent neural network module 304 based on the plurality of first parameter values corresponding to the plurality of first parameters 303, that is, the trained parameters of the convolutional neural network module 302.

[0051] In step S1308, the processor 301 further obtains a plurality of second sample images of each video from each video. The processor 301 trains the recurrent neural network module 304 based on the plurality of first parameter values and all the plurality of second sample images obtained from each video to obtain a plurality of second parameter values corresponding to the plurality of second parameters 305.

[0052] It should be noted that before the processor 301 trains the first synthetic network 404 composed of the convolutional neural network module 302 and the classification module 306, the first parameters 303 of the convolutional neural network module 302 need to be pre-stored with initial values. The processor 301 can randomly set the initial values of the first parameters based on the function of generating random values provided by the existing program language suite. The processor 301 can also use the trained parameters trained in other similar tasks as the initial values of the first parameters 303 of the convolutional neural network module 302.

[0053] Figure 7This is a schematic diagram illustrating the architecture of a classification module 306 according to some embodiments of the present disclosure. The classification module 306 includes an input layer 801, a hidden layer 802, and an output layer 803. The input layer 801 includes 1024 input neurons. x1, x2…x 1024 The input signal is used. The output layer 803 includes two output neurons and a normalized exponential function (softmax) module 804. The normalized exponential function (softmax) module 804 ensures that the sum of the outputs y1 and y2 is 1 and both are between 0 and 1. y1 represents the probability that the input belongs to the first category, and y2 represents the probability that the input belongs to the second category. In this embodiment, before the processor 301 trains the first synthesized network 404 synthesized by the convolutional neural network module 302 and the classification module 306, the initial value of the third parameter of the classification module 306 is randomly set.

[0054] Figure 5 This is a schematic diagram illustrating the operation of a neural network training system 300 according to some embodiments of the present disclosure. Figure 13 The following is a flowchart illustrating a neural network training method based on some embodiments of this disclosure. Please also refer to... Figure 3 , Figure 5 as well as Figure 13 .like Figure 13 As illustrated, step S1308 further includes steps S1401 to S1411. In step S1401, the processor 301 selects an unselected video from the plurality of videos as the current video. In step S1402, the processor 301 performs multiple time points t0, t1, t2, t3, t4, t5, t6…t on the current video. n , obtain second sampled images 500, 501, 502, 503, 504, 505, 506...50n from the current video.

[0055] In step S1403, the processor 301 determines the time points t0, t1, t2, t3, t4, t5, t6…t based on the given time points. nIn the order of the first number, the processor 301 selects a first number of the second sample images from the plurality of second sample images 500, 501, 502, 503, 504, 505, 506, …, 50n as a plurality of sample images. It should be noted that the definition of "in order" in this embodiment is that each time a new second sample image is added and an old second sample image is removed compared with the previous time. For example, the first number is 5. The plurality of sample images selected by the processor 301 in the first time are the second sample images 500, 501, 502, 503, and 504. The plurality of sample images selected by the processor 301 in the second time are the second sample images 501, 502, 503, 504, and 505. The plurality of sample images selected by the processor 301 in the third time are the second sample images 502, 503, 504, 505, and 506. In this way, the plurality of sample images are obtained.

[0056] In step S1404, the processor 301 selects an unselected sample image from the plurality of sample images as a selected sample image. In step S1405, the processor 301 performs pixel size adjustment on the selected sample image to obtain a corresponding third image, wherein the pixel size of the third image is the first pixel size, and in this embodiment, the first pixel size is 224x224x3. In step S1406, the processor 301 performs image cropping on the selected sample image to obtain at least one second partial image, and obtains a corresponding fourth image based on the at least one second partial image, wherein the pixel size of the fourth image is the first pixel size 224x224x3. In step S1407, the processor 301 repeats steps S1404, S1405, and S1406 until all the plurality of sample images are selected.

[0057] After step S1407, each of the plurality of sample images has a third image and a fourth image. In step S1408, the processor 301 uses the convolutional neural network module 302 to encode all the third images and the fourth images corresponding to each sample image into a sample feature matrix corresponding to the plurality of sample images.

[0058] In this embodiment, the plurality of sample images are the second sample images 500, 501, 502, 503, and 504. The processor 301 repeats steps S1404, S1405, and S1406 to obtain the third image and the fourth image corresponding to each second sample image 500, 501, 502, 503, and 504. In step S1408, the processor 301 uses the convolutional neural network module 302 to encode all the third images and the fourth images corresponding to the second sample images 500, 501, 502, 503, and 504 into a sample feature matrix 60, i.e. Figure 5The matrix composed of vectors 600-604 is illustrated. Vector 600 is the third image and the fourth image of the second sample image 500 encoded by the processor 301 using the convolutional neural network module 302, and so on. The sample feature matrix 61 is a matrix composed of vectors 601-605, and the sample feature matrix 62 is a matrix composed of vectors 602-606.

[0059] In step S1409, the processor 301 sets the classification corresponding to the sample feature matrix and the plurality of sample images as a second training sample corresponding to the plurality of sample images. For example, taking the selected plurality of sample images as the second sample images 500, 501, 502, 503, and 504 as an example, the second sample images 500, 501, 502, 503, and 504 are all from the same video, so the classification of this video is taken as the classification of the second sample images 500, 501, 502, 503, and 504. And the sample feature matrix 60 and this classification are set as the second training sample of the sample images composed of the second sample images 500, 501, 502, 503, and 504.

[0060] In step S1410, the processor 301 repeats steps S1401 to S1409 until all videos are selected. After step S1410, the processor 301 obtains a plurality of second training samples. In step S1411, the processor 301 trains the recurrent neural network module 304 based on the second training samples corresponding to the plurality of sample images of the plurality of second sample images of all videos to obtain the second parameter values of the plurality of second parameters 305.

[0061] In some embodiments of the present disclosure, step S1406 further includes that the processor 301 performs image cropping at a second corresponding position of each sample image to obtain a second cropped image with a first pixel size of 224x224x3. The processor 301 takes this second cropped image as at least one second partial image and the fourth image, wherein the second corresponding position is the same as the first corresponding position.

[0062] In some embodiments of the present disclosure, step S1406 further includes that the processor 301 performs image cropping at a plurality of second corresponding positions of each sample image to obtain a plurality of second cropped images with a first pixel size of 224x224x3. The processor 301 takes the plurality of second cropped images as at least one second partial image. The processor 301 further averages the plurality of second cropped images to obtain the fourth image, wherein the plurality of second corresponding positions are the same as the plurality of first corresponding positions.

[0063] As the foregoing reasons, in an embodiment of the present disclosure, the recurrent neural network module 304 is selected as a long short-term memory network.

[0064] It is worth noting that, since in the foregoing embodiment, the convolutional neural network module 302 is first trained, and then the recurrent neural network module 304 is trained according to the trained convolutional neural network module 302, the two are trained separately. Therefore, when the processor 301 finds that the prediction accuracy is not as expected when performing the test, the processor 301 can receive other videos again, and according to steps S1401-S1411, the recurrent neural network module 304 is retrained again using the first parameter value of the trained convolutional neural network module 302 to improve the prediction accuracy.

[0065] Figure 6 A structural schematic diagram of an electronic device 700 according to some embodiments of the present disclosure is shown. As shown in the hardware layer, the electronic device 700 includes a processor 701, an internal memory 702, and a non-volatile memory 703. The internal memory 702 is, for example, a random-access memory (RAM). The non-volatile memory is, for example, at least one disk memory or the like. Of course, the electronic device 700 can also include other hardware required by functions. Figure 6

[0066] The internal memory 702 and the non-volatile memory 703 are used to store programs, which can include program codes including computer operation instructions. The internal memory 702 and the non-volatile memory 703 provide instructions and data to the processor 701. The processor 701 reads the corresponding computer program from the non-volatile memory 703 into the internal memory 702 and then runs. The processor 701 is specifically configured to execute the steps described. Figures 8 to 13

[0067] ​​The processor 701 can be an integrated circuit chip having a processing capability of signals. In implementation, the methods and steps disclosed in the foregoing embodiments can be completed by integrated logic circuits or instructions in the form of software in the processor 701. The processor 701 can be a general-purpose processor, including a central processing unit (CPU), a tensor processing unit (TPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, which can implement or execute the methods and steps disclosed in the foregoing embodiments.

[0068] The embodiments of the present disclosure further provide a computer-readable storage medium storing at least one instruction, which, when executed by the processor 701 of the electronic device 700, can enable the processor 701 of the electronic device 700 to perform the methods and steps disclosed in the foregoing embodiments.

[0069] Examples of the storage medium of the computer include, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, or other internal memory technology, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), or other optical storage, a magnetic cassette tape, a magnetic tape disk storage, or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, the computer-readable medium does not include transitory media, such as modulated data signals and carriers.

[0070] Based on the above, some embodiments of the present disclosure provide a video classification system, a video classification method, a neural network training system, and a neural network training method. By using a convolutional neural network module and a recurrent neural network module trained in advance on images including two different types of information in video capture images, the multiple sample pictures of the real-time video are classified, and the classification result of the video can be quickly obtained.

Claims

1. A video classification system, characterized by, comprises: a processor configured to obtain a video; a convolutional neural network module having a plurality of trained first parameters; and a recurrent neural network module having a plurality of trained second parameters; wherein the processor is configured to perform the following steps: (a) selecting a present time point according to a time interval, sampling the video according to the present time point to obtain a sampled image of the present time point; (b) performing pixel size adjustment on the sampled image of the present time point to obtain a corresponding first image, wherein a pixel size of the first image is a first pixel size, and the first pixel size of the first image is smaller than a pixel size of the sampled image; (c) performing image cropping on the sampled image of the present time point to obtain at least one partial image, and obtaining a corresponding second image based on the at least one partial image, wherein a pixel size of the second image is the first pixel size; (d) using the convolutional neural network module to encode the first image and the second image of the sampled image of the present time point into a feature vector corresponding to the present time point; (e) sequentially merging the feature vector corresponding to the present time point and a plurality of past feature vectors corresponding to a plurality of past time points into a feature matrix; and (f) obtaining a classification of the video based on the recurrent neural network module and the feature matrix. The processor performs image cropping on a corresponding position of the sampled image to obtain a cropped image having the first pixel size, and uses the cropped image as the at least one partial image and the second image.

2. The video classification system of claim 1, wherein, The processor performs image cropping on a plurality of corresponding positions of the sampled image to obtain a plurality of cropped images having the first pixel size, and uses the plurality of cropped images as the at least one partial image. The processor further averages the plurality of cropped images to obtain the second image.

3. The video classification system of claim 1, wherein, The processor uses the convolutional neural network module to encode the first image of the sampled image into a first partial feature vector, uses the convolutional neural network module to encode the second image of the sampled image into a second partial feature vector, and combines the first partial feature vector and the second partial feature vector to obtain the feature vector corresponding to the present time point.

4. The video classification system of claim 1, wherein, 5. A video classification method suitable for a video classification system, the video classification method being performed by a processor, the video classification system comprising: the processor configured to obtain a video; a convolutional neural network module having a plurality of trained first parameters; and a recurrent neural network module having a plurality of trained second parameters; wherein the video classification method comprises the following steps: (a) selecting a present time point according to a time interval, sampling the video according to the present time point to obtain a sampled image of the present time point; (b) performing pixel size adjustment on the sampled image of the present time point to obtain a corresponding first image, wherein a pixel size of the first image is a first pixel size, and the first pixel size of the first image is smaller than a pixel size of the sampled image; ​ ​ (c) performing image cropping on the sampled image at the present time point to obtain a corresponding at least one partial image, and obtaining a corresponding second image based on the at least one partial image, wherein a pixel size of the second image is the first pixel size; (d) using the convolutional neural network module, encoding the first image and the second image of the sampled image at the present time point into a feature vector corresponding to the sampled image at the present time point; (e) sequentially merging the feature vector corresponding to the present time point and a plurality of past feature vectors corresponding to a plurality of past time points into a feature matrix; and (f) obtaining a classification of the video based on the recurrent neural network module and the feature matrix.

6. A neural network training system, comprising: The processor is configured to perform the following steps: (a) obtaining a plurality of first sampled images from the plurality of videos and the plurality of training images; (b) selecting an unselected image from the plurality of first sampled images as a current image; (c) performing pixel size adjustment on the current image to obtain a corresponding first image, wherein a pixel size of the first image is a first pixel size, and the first pixel size of the first image is smaller than a pixel size of the current image; (d) performing image cropping on the current image to obtain a corresponding at least one first partial image, and obtaining a corresponding second image based on the at least one first partial image, wherein a pixel size of the second image is the first pixel size; (e) setting the classification corresponding to the first image, the second image, and the current image as a first training sample corresponding to the current image; (f) repeating steps (b), (c), (d), and (e) until all of the plurality of first sampled images are selected; (g) using all of the first training samples corresponding to each first sampled image, training a first synthesized network composed of the convolutional neural network module and the classification module to obtain a plurality of first parameter values corresponding to the plurality of first parameters; and (h) based on the plurality of first parameter values and a plurality of second sampled images obtained from each video, training the recurrent neural network module to obtain a plurality of second parameter values corresponding to the plurality of second parameters. The processor is configured to perform step (h) further comprising: (h1) selecting an unselected video from the plurality of videos as a current video; (h2) from the current video, obtaining the plurality of second sampled images at a plurality of time points of the current video; (h3) sequentially selecting a first number of second sampled images from the plurality of second sampled images as a plurality of sample images according to an order of the plurality of time points; (h4) selecting an unselected sample image from the plurality of sample images as a selected sample image; ​ 7. The neural network training system of claim 6, wherein, ​ ​ ​ ​ ​ (h5) performing pixel size adjustment on the selected sample image to obtain a corresponding third image, wherein a pixel size of the third image is the first pixel size; (h6) performing image cropping on the selected sample image to obtain a corresponding at least one second partial image, and based on the at least one second partial image, obtaining a corresponding fourth image, wherein a pixel size of the fourth image is the first pixel size; (h7) repeating steps (b), (c) and (d) until all of the plurality of sample images are selected; (h8) using the convolutional neural network module to encode the third image and the fourth image corresponding to each sample image into a sample feature matrix corresponding to the plurality of sample images; (h9) setting the sample feature matrix and the classification corresponding to the plurality of sample images as a second training sample corresponding to the plurality of sample images; (h10) repeating steps (h1) to (h9) until all of the plurality of videos are selected; and (h11) training the recurrent neural network module based on the second training sample corresponding to the plurality of sample images of the plurality of second sample images of all of the plurality of videos to obtain the plurality of second parameter values of the plurality of second parameters.

8. The neural network training system of claim 7, wherein, The processor performs image cropping at a first corresponding position of each first sample image to obtain a first cropped image having a pixel size of the first pixel size, and uses the first cropped image as the at least one first partial image and the second image; and the processor performs image cropping at a second corresponding position of each sample image to obtain a second cropped image having a pixel size of the first pixel size, and uses the second cropped image as the at least one second partial image and the fourth image, wherein the second corresponding position has the same position as the first corresponding position.

9. The neural network training system of claim 7, wherein, The processor performs image cropping at a plurality of first corresponding positions of each first sample image to obtain a plurality of first cropped images having a pixel size of the first pixel size, and uses the plurality of first cropped images as the at least one first partial image, and the processor further averages the plurality of first cropped images to obtain the second image; and the processor performs image cropping at a plurality of second corresponding positions of each sample image to obtain a plurality of second cropped images having a pixel size of the first pixel size, and uses the plurality of second cropped images as the at least one second partial image, and the processor further averages the plurality of second cropped images to obtain the fourth image, wherein the plurality of second corresponding positions have the same positions as the plurality of first corresponding positions.

10. A neural network training method suitable for a neural network training system, executed by a processor, the neural network training system comprising: the processor configured to obtain a plurality of videos, a plurality of training images, and a classification corresponding to each video and corresponding to each training image; a convolutional neural network module having a plurality of first parameters; a recurrent neural network module having a plurality of second parameters; and a classification module having a plurality of third parameters; characterized in that the neural network training method comprises the following steps: ​ (a) obtaining a plurality of first sample images from the plurality of videos and the plurality of training images; (b) selecting an unselected image from the plurality of first sample images as a current image; (c) performing pixel size adjustment on the current image to obtain a corresponding first image, wherein a pixel size of the first image is a first pixel size, and the first pixel size of the first image is smaller than a pixel size of the current image; (d) performing image cropping on the current image to obtain at least one first partial image, and based on the at least one first partial image, obtaining a corresponding second image, wherein a pixel size of the second image is the first pixel size; (e) setting a classification corresponding to the first image, the second image and the current image as a first training sample corresponding to the current image; (f) repeating steps (b), (c), (d) and (e) until all of the plurality of first sample images are selected; (g) using all of the first training samples corresponding to each of the first sample images, training a first synthesized network synthesized by the convolutional neural network module and the classification module to obtain a plurality of first parameter values corresponding to the plurality of first parameters; and (h) based on the plurality of first parameter values and all of a plurality of second sample images obtained from each of the videos, training the recurrent neural network module to obtain a plurality of second parameter values corresponding to the plurality of second parameters.

Images