Data processing methods, apparatus, storage media and computer equipment

Abstract

This invention discloses a data processing method, apparatus, storage medium, and computer device. The method includes: acquiring multi-frame image data of a video; and identifying the video scene by inputting the multi-frame image data into a video recognition model. The video recognition model includes a video feature module, which extracts image features from the temporally correlated multi-frame image data. The video recognition model is trained using multiple sets of data, each set including: multi-frame image data of the video and the video scene of the video. This invention solves the technical problem of inaccurate video scene identification.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a data processing method, apparatus, storage medium, and computer equipment. Background Technology

[0002] With the development of attachment technology, artificial intelligence (AI) has been applied to various fields. For example, AI can be applied to various recognition fields, such as using recognition models to identify objects or instructions. However, in related technologies, the use of AI models for recognition is mostly limited to identifying static content. For instance, scene recognition often involves using a recognition model to identify images. But when recognizing images, because images are static and have limited features, they cannot fully represent the content of the entire scene, thus limiting the accuracy of recognition. Therefore, related technologies suffer from the problem of inaccurate recognition of video scenes.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, storage medium, and computer device to at least solve the technical problem of inaccurate video scene identification.

[0005] According to one aspect of the present invention, a data processing method is provided, comprising: acquiring multi-frame image data of a video; and identifying a video scene of the video by inputting the multi-frame image data into a video recognition model, wherein the video recognition model includes a video feature module, the video feature module being used to extract image features of the multi-frame image data that are temporally correlated, wherein the video recognition model is trained by multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video.

[0006] Optionally, the video feature module includes a global feature submodule and an object local feature submodule. Identifying the video scene of the video by inputting the multi-frame image data into the video recognition model includes: extracting global video features from each frame of the multi-frame image data using the global feature submodule, wherein the global video features include features exceeding a predetermined dimension for each frame of the image data, and temporal information between the multi-frame image data; extracting object local features from each frame of the multi-frame image data using the object local feature submodule; and identifying the video scene of the video based on the global video features and the object local features.

[0007] Optionally, the video recognition model further includes an attention module, wherein recognizing the video scene of the video based on the video global features and the object local features includes: assigning a first weight coefficient to the object local features through the attention module according to the video global features, and assigning a second weight coefficient to the video global features of each frame of image data respectively; updating the object local features according to the first weight coefficient to obtain updated object local features, and updating the video global features of each frame of image data according to the second weight coefficient to obtain updated video global features of each frame of image data; and recognizing the video scene of the video based on the obtained updated object local features and the updated video global features of each frame of image data.

[0008] Optionally, the video recognition model further includes a classifier, wherein identifying the video scene of the video based on the obtained updated local object features and the updated global video features of each frame of image data includes: processing the updated local object features and the updated global video features of each frame of image data through an attention module to obtain video output features; processing the video output features using a classifier; and determining the video scene of the video based on the classification result of the classifier.

[0009] Optionally, the video recognition model further includes a classifier, wherein identifying the video scene of the video based on the video output features includes: using the classifier to classify the updated local features of the object and the updated global video features of each frame of image data, respectively obtaining a classification result corresponding to the updated local features of the object and a classification result corresponding to the updated global video features of each frame of image data, wherein each classification result corresponds to a video scene; and determining the video scene of the video based on the classification result corresponding to the updated local features of the object and the classification result corresponding to the updated global video features of each frame of image data.

[0010] Optionally, multi-frame image data of a video can be obtained by at least one of the following methods: acquiring consecutive multi-frame image data in the video; or sampling the video multiple times according to a fixed frame interval to obtain multi-frame image data in the video.

[0011] Optionally, the video includes at least one of the following: live video, recorded video.

[0012] Optionally, the multi-frame image data is 16-frame image data.

[0013] According to another aspect of the present invention, a data processing method is also provided, comprising: displaying a video on an interactive interface; receiving a request message on the interactive interface requesting the acquisition of a video scene of the video; responding to the request message, acquiring multi-frame image data of the video, and identifying the video scene of the video by inputting the multi-frame image data into a video recognition model, wherein the video recognition model includes a video feature module, the video feature module being used to extract image features of the multi-frame image data that are temporally correlated, wherein the video recognition model is trained through multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video; and displaying the video scene on the interactive interface.

[0014] According to another aspect of the present invention, a data processing method is also provided, comprising: broadcasting a video live on a live streaming interface; displaying a video scene of the video on the live streaming interface, wherein the video scene of the video is obtained by recognizing multi-frame image data of the video using a video recognition model, the video recognition model including a video feature module, the video feature module being used to extract image features of the multi-frame image data that are temporally related, wherein the video recognition model is trained through multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video.

[0015] According to another aspect of the present invention, a data processing method is also provided, comprising: a front-end client playing a video; the front-end client sending a request message for requesting the display of a video scene of the video to a back-end server; the front-end client receiving the video scene fed back by the back-end server and displaying the video scene, wherein the video scene is obtained by the back-end server based on a video recognition model applied to multi-frame image data of the video, the video recognition model including a video feature module, the video feature module being used to extract image features of the multi-frame image data that are temporally related, wherein the video recognition model is trained through multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video.

[0016] According to another aspect of the present invention, a data processing apparatus is also provided, comprising: an acquisition module for acquiring multi-frame image data of a video; and a first recognition module for recognizing a video scene of the video by inputting the multi-frame image data into a video recognition model, wherein the video recognition model includes a video feature module for extracting image features of the multi-frame image data that are temporally correlated, and wherein the video recognition model is trained using multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video.

[0017] According to another aspect of the present invention, a data processing apparatus is also provided, comprising: a first display module for displaying a video on an interactive interface; a receiving module for receiving a request message on the interactive interface requesting the acquisition of a video scene of the video; a second recognition module for responding to the request message, acquiring multi-frame image data of the video, and recognizing the video scene of the video by inputting the multi-frame image data into a video recognition model, wherein the video recognition model includes a video feature module, the video feature module being used to extract image features of the multi-frame image data that are temporally correlated, wherein the video recognition model is trained using multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video; and a second display module for displaying the video scene on the interactive interface.

[0018] According to another aspect of the present invention, a data processing apparatus is also provided, comprising: a live streaming module for live streaming video on a live streaming interface; and a third display module for displaying a video scene of the video on the live streaming interface, wherein the video scene of the video is obtained by recognizing multi-frame image data of the video using a video recognition model, the video recognition model including a video feature module, the video feature module being used to extract image features of the multi-frame image data that are temporally related, wherein the video recognition model is trained through multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video.

[0019] According to another aspect of the present invention, a data processing apparatus is also provided, comprising: a playback module for a front-end client to play a video; a sending module for the front-end client to send a request message for requesting the display of a video scene of the video to a back-end server; and a fourth display module for the front-end client to receive the video scene fed back by the back-end server and display the video scene, wherein the video scene is obtained by the back-end server based on a video recognition model applied to multi-frame image data of the video, the video recognition model including a video feature module, the video feature module being used to extract image features of the multi-frame image data that are temporally correlated, wherein the video recognition model is trained through multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video.

[0020] According to another aspect of the present invention, a storage medium is also provided, comprising: the storage medium including a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the data processing method described in any one of the above embodiments.

[0021] According to another aspect of the present invention, a computer device is also provided, comprising: a memory and a processor, the memory storing a computer program; the processor being configured to execute the computer program stored in the memory, wherein the computer program, when executed, causes the processor to perform any of the data processing methods described above.

[0022] According to one aspect of the present invention, a data processing method is provided, comprising: collecting multiple sets of data, wherein the multiple sets of data respectively include: multi-frame image data of a video and a video scene of the video; extracting image features of the multi-frame image data that are temporally related from the multiple sets of data respectively, and performing machine learning training on the multiple sets of data to obtain a video recognition model.

[0023] According to another aspect of the present invention, a data processing apparatus is provided, comprising: an acquisition module for acquiring multiple sets of data, wherein the multiple sets of data respectively include: multi-frame image data of a video and a video scene of the video; and a training module for performing machine learning training on the multiple sets of data by extracting image features of the multi-frame image data that are temporally related from the multiple sets of data, thereby obtaining a video recognition model.

[0024] In this embodiment of the invention, a method of acquiring multi-frame image data of a video is adopted. By inputting the multi-frame image data into a video recognition model including a video feature module, wherein the video feature module is used to extract image features of multi-frame image data that are related in time sequence, the purpose of recognizing the video scene of the video is achieved, thereby solving the technical problem of being unable to accurately recognize the video scene. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0026] Figure 1 It is a hardware structure block diagram of a computer terminal used to implement data processing methods;

[0027] Figure 2 This is a flowchart of a data processing method according to Embodiment 1 of the present invention;

[0028] Figure 3 This is a flowchart of the second data processing method provided in Embodiment 1 of the present invention;

[0029] Figure 4 This is a flowchart of the third data processing method provided in Embodiment 1 of the present invention;

[0030] Figure 5This is a flowchart of the fourth data processing method provided in Embodiment 1 of the present invention;

[0031] Figure 6 This is a flowchart of the data processing method five provided in Embodiment 1 of the present invention;

[0032] Figure 7 This is a structural block diagram of a data processing apparatus according to Embodiment 2 of the present invention;

[0033] Figure 8 This is a structural block diagram of the data processing device 2 provided in Embodiment 3 of the present invention;

[0034] Figure 9 This is a structural block diagram of the data processing device three provided in Embodiment 4 of the present invention;

[0035] Figure 10 This is a structural block diagram of the data processing device four provided in Embodiment 5 of the present invention;

[0036] Figure 11 This is a structural block diagram of the data processing apparatus five provided in Embodiment 5 of the present invention;

[0037] Figure 12 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] Example 1

[0041] According to an embodiment of the present invention, a data processing method embodiment is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] The data processing method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0043] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the data processing method of the aforementioned application. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0046] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0047] Under the aforementioned operating environment, this application provides the following: Figure 2 The data processing method shown. Figure 2 This is a flowchart of a data processing method according to Embodiment 1 of the present invention. Figure 2 As shown, the process includes the following steps:

[0048] Step S202: Acquire multi-frame image data from the video;

[0049] Step S204: By inputting multi-frame image data into the video recognition model, the video scene of the video is identified. The video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are related in time. The video recognition model is trained by multiple sets of data, which include multi-frame image data of the video and the video scene of the video.

[0050] By employing the above steps and acquiring multi-frame image data from a video, and inputting this multi-frame image data into a video recognition model that includes a video feature module, the video feature module extracts image features from the temporally related multi-frame image data. This achieves the goal of identifying video scenes, thus realizing the technical effect of accurately identifying video scene types based on video content and solving the technical problem of not being able to accurately identify video scenes based on video content.

[0051] As an optional embodiment, the aforementioned multiple sets of data may include multiple frames of image data from a video and the video scene itself. This could mean that all sets of data include both multiple frames of image data from the video and the video scene; or that most sets of data include both multiple frames of image data from the video and the video scene; or that more than a predetermined number of sets of data include both multiple frames of image data from the video and the video scene. These multiple sets of data are used for model training, and the number and content of the multiple sets of data can be flexibly selected according to the requirements of model training. Therefore, to a certain extent, based on the specific requirements of model training, the aforementioned multiple sets of data including both multiple frames of image data from the video and the video scene may mean that at least one set of data in the multiple sets includes both multiple frames of image data from the video and the video scene.

[0052] As an optional embodiment, the video feature module includes a global feature submodule and an object local feature submodule. After inputting multi-frame image data into the video recognition model, the global feature submodule extracts the global video features of each frame of the multi-frame image data. The global video features include features of each frame of the image data exceeding a predetermined dimension, as well as temporal information between the multi-frame image data. The object local feature submodule extracts the object local features of each frame of the multi-frame image data. Then, based on the global video features and the object local features, the video scene of the video is identified.

[0053] As an optional embodiment, the aforementioned global video features include features exceeding a predetermined dimension in each frame of image data. The predetermined dimension can be flexibly selected based on the video scene recognition requirements. For example, when scene recognition requirements are high, the predetermined dimension can be higher, such as 2048 dimensions or more. When scene recognition requirements are moderate, the predetermined dimension can be lower, such as 1024 dimensions or less.

[0054] As an optional implementation, the video global feature submodule and the object local feature submodule can be neural networks, pre-trained. For example, the input to the video global feature network can be 16 uniformly sampled frames from the video. In this case, the network outputs 16 2048-dimensional vectors corresponding to the 16 frames, which are the 16 video global features. Furthermore, the video global feature submodule can use a Temporal-Shfit-Module (TSM) RestNet network as the backbone network for extracting video global features. The Temporal-Shfit-Module, based on a 2D convolutional network, can perform displacement operations on features from different channels in the time dimension, thereby establishing temporal relationships between multiple video global image features. Therefore, the video global features extracted using the Temporal-Shfit-Module not only include the image information in each frame of image data but also the temporal relationship between that frame and the other acquired frames.

[0055] For example, while capturing the nth frame of a video, two frames are captured one second before and after it. The video global features of this frame are extracted from these three frames and labeled as video global feature 1. Then, two frames are captured two seconds before and after the nth frame, and video global features 2 are extracted from the nth frame based on these two frames. Although the nth frame remains unchanged and the extracted video global features are all from the nth frame, the timing and content of the context video images used for the nth frame are different, resulting in different video global features 1 and 2 for the nth frame.

[0056] The object local feature submodule can extract local object features from each frame of multi-frame image data. Video scenes are highly complex, with each frame containing numerous objects. Therefore, the object local feature submodule first uses a detector to outline the objects in each frame, and then extracts local object features based on these outlined images. Not all objects in an image are crucial for identifying the scene type; therefore, the object local feature submodule selects a predetermined number of object local features with the highest confidence from the multiple object local features corresponding to each frame as input data for subsequent processes. For example, after sampling 16 frames from a video, the object local feature submodel processes each frame, selecting the 12 object local features with the highest confidence in each frame. Thus, a total of 16*12=192 object local features are selected from the 16 frames through the object local feature submodel.

[0057] As an optional embodiment, the video recognition model includes an attention module. The process of identifying the video scene based on global video features and local object features includes: assigning a first weight coefficient to the local object features using the attention module, and assigning a second weight coefficient to the global video features of each frame of image data; updating the local object features based on the first weight coefficient to obtain updated local object features, and updating the global video features of each frame of image data based on the second weight coefficient to obtain updated global video features of each frame of image data; and identifying the video scene based on the updated local object features and the updated global video features of each frame of image data.

[0058] Through the above optional embodiments, the attention module in the video recognition model can be used to match weights for the global video features and local object features extracted by the video feature module, thereby optimizing the results of the subsequent scene recognition process. In this embodiment of the invention, the attention module can play at least two roles: first, it associates and fuses global video features with local object features. The attention module can assign weights to local object features based on the global video features, thereby effectively utilizing the local object features that contribute the most to scene recognition and eliminating interference from local object features that are irrelevant to scene recognition or contribute less; second, it assigns weights to global video features across multiple frames. Based on the contextual relationship of multiple frames, weights are assigned to global video features of different frames according to the attention mechanism, effectively utilizing high-resolution single-frame features and eliminating noisy interfering frame features.

[0059] The attention module can generate a query vector q = Q(a) for a sample a, and a base key vector and value vector k = K(b) and v = V(b) for the context b. Then, it calculates the inner product between the query and the key, and uses the softmax function to transform it into a probability distribution, i.e., a weight distribution: w = softmax(q·k). Then, the weight distribution is applied to the value vector, and the value is summed in weight. The final result is added to sample a in the form of residuals: a' = a + w·v, where a' is the feature updated by the attention module.

[0060] As an optional implementation, the video recognition model also includes a classifier. The classifier can participate in the recognition of video scenes in various ways. Examples are given below.

[0061] For example, the updated local features of objects and the updated global features of each frame of image data can be processed by the attention module to obtain the video output features; then, a classifier is used to process the video output features, and the video scene is determined based on the classification result of the classifier. Specifically, in this embodiment of the invention, a multi-layer transformer (Transformer) structure, which is widely used in the field of natural language processing, can be used as the network model of the attention module. The Transformer structure takes the feature sequence as input, processes the self-attention mechanism internally, and outputs the feature sequence after the self-attention mechanism. At the input end, the [CLS] classification node, the global video features (i.e., 16 2048-dimensional features) and the local object features (the 12 local object features with the highest confidence in each of the 16 frames of images, for a total of 192 2048-dimensional features) can be concatenated into a sequence and reduced to 768 dimensions. The frame number is used as the position embedding, and the global and local features are used as two types of embedding. After being summed with the features, the result is input into the attention module composed of 3 layers of Transformers. At the output end, we take the feature corresponding to the [CLS] classification node as the final video output feature. Then, the video output features of the attention module are input into the classifier to obtain the classification result and determine the video scene type of the video.

[0062] In addition, a classifier can be used to classify the updated local features of the object and the updated global video features of each frame of image data, respectively obtaining the classification results corresponding to the updated local features of the object and the classification results corresponding to the updated global video features of each frame of image data. Each classification result corresponds to a video scene. Based on the classification results corresponding to the updated local features of the object and the classification results corresponding to the updated global video features of each frame of image data, the video scene of the video is determined.

[0063] It should be noted that both the method of first synthesizing features and then classifying the synthesized features, and the method of first classifying features and then synthesizing the classification results, are capable of effectively recognizing video scenes. The appropriate method can be chosen flexibly based on specific needs.

[0064] As an optional embodiment, multi-frame image data of a video can be obtained by acquiring continuous multi-frame image data from the video or by sampling the video multiple times according to a fixed frame interval. The method used to acquire multi-frame image data can be determined according to the application scenario or the characteristics of the video.

[0065] As an optional embodiment, the video to be identified can be a live video or a recorded video. The above method can be used to identify scene types in recorded videos, or it can be used to collect image data in live videos in real time, and then identify scenes in live videos.

[0066] For example, with the rise of live video streaming, e-commerce live streaming has experienced explosive growth. In the distribution and processing of live content, the live stream start-up scene information includes information about the live stream's environment, providing a feature dimension for user searches and platform recommendations. Since real-time live video streams contain rich information, we hope to accurately identify the live stream start-up scene based on the video content. However, there is no manually labeled information available for identifying the live stream start-up scene, and currently, there is no method to accurately identify the live stream scene based on video content. Because the scenes in live videos are very complex, and the streamer's activities may affect the quality of the video images, traditional methods for identifying image scenes based on single-frame static images are difficult to perform well in identifying scene categories in videos. The method provided in this invention considers multi-frame image data of the video and the temporal information between these frames, which can effectively handle rapid changes in video images, eliminate noisy frames and interference from irrelevant objects in the image, and accurately identify the video scene based on more effective information.

[0067] In this embodiment of the invention, a data processing method is also provided. Figure 3 This is a flowchart of the second data processing method according to Embodiment 1 of the present invention, as follows: Figure 3 As shown, the process includes the following steps:

[0068] Step S302: Display the video on the interactive interface;

[0069] Step S304: A request message for obtaining the video scene is received on the interactive interface;

[0070] Step S306: Respond to the request message, acquire multi-frame image data of the video, and identify the video scene of the video by inputting the multi-frame image data into the video recognition model. The video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are related in time. The video recognition model is trained by multiple sets of data, which include: multi-frame image data of the video and the video scene of the video.

[0071] Step S308: Display the video scene on the interactive interface.

[0072] Through the above steps, by displaying the video on the interactive interface and receiving and responding to request messages for retrieving the video scene on the interactive interface, the video scene is identified by acquiring multi-frame image data of the video and inputting the multi-frame image data into the video recognition model. This achieves the goal of displaying the video scene on the interactive interface, thus realizing the technical effect of displaying the video scene on the interactive interface based on the video displayed on the interactive interface and the request messages for retrieving the video scene received by the interactive interface. This solves the technical problem of not being able to accurately return the video scene based on the video content and the request from the interactive interface.

[0073] In this embodiment of the invention, a data processing method is also provided. Figure 4 This is a flowchart of the third data processing method according to Embodiment 1 of the present invention, as follows: Figure 4 As shown, the process includes the following steps:

[0074] Step S402: Live stream video on the live streaming interface;

[0075] Step S404: Display the video scene of the video on the live streaming interface. The video scene is obtained by recognizing the multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module, which is used to extract image features of the multi-frame image data that are related in time. The video recognition model is trained by multiple sets of data, which include the multi-frame image data of the video and the video scene of the video.

[0076] Through the above steps, by using live video on the live streaming interface and employing a video recognition model to identify multi-frame image data of the video, the goal of displaying the video scene on the live streaming interface is achieved. This realizes the technical effect of displaying the video scene on the live streaming interface based on the live video, and thus solves the technical problem of not being able to display the video scene on the live streaming interface based on the live video.

[0077] In this embodiment of the invention, a data processing method is also provided. Figure 5 This is a flowchart of the fourth data processing method according to Embodiment 1 of the present invention, as follows: Figure 5 As shown, the process includes the following steps:

[0078] Step S502: The front-end client plays the video;

[0079] Step S504: The front-end client sends a request message for the video scene to be displayed to the back-end server.

[0080] In step S506, the front-end client receives the video scene fed back by the back-end server and displays the video scene. The video scene is obtained by the back-end server based on the multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are related in time. The video recognition model is trained through multiple sets of data, which include: multi-frame image data of the video and the video scene of the video.

[0081] Through the above steps, the interaction between the front-end client and the back-end server allows the front-end client to send a request message to the back-end server to request the display of the video scene of the video being played on the front-end client. The back-end server then uses a video recognition model to identify the video scene based on the multi-frame image data of the video and feeds it back to the front-end client. This achieves the goal of displaying the video scene on the front-end client, thus realizing the technical effect of displaying the video scene on the front-end client based on the video being played on the front-end client. This solves the technical problem of not being able to display the video scene on the front-end client based on the video being played on the front-end client.

[0082] In this embodiment of the invention, a data processing method is also provided. Figure 6 This is a flowchart of data processing method five according to embodiment 1 of the present invention, as follows: Figure 6 As shown, the process includes the following steps:

[0083] Step S602: Collect multiple sets of data, including: multiple frames of image data from the video and the video scene of the video;

[0084] Step S604: By extracting image features from multiple frames of image data that are temporally related in multiple sets of data, machine learning training is performed on the multiple sets of data to obtain a video recognition model.

[0085] Through the above steps, by collecting multiple sets of data and extracting image features from multiple frames of image data that are temporally related in the multiple sets of data, and by performing machine learning training on the multiple sets of data to obtain a video recognition model, the goal of training a model for video to intelligently identify video scenes is achieved. This provides a basic technical effect for accurately and efficiently identifying video scenes based on videos, and solves the technical problem of low efficiency in identifying video scenes based on static images in related technologies.

[0086] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the data processing method according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0088] Example 2

[0089] According to an embodiment of the present invention, a data processing apparatus for implementing the above-described data processing method is also provided. Figure 7 This is a structural block diagram of a data processing apparatus according to Embodiment 2 of the present invention, as shown below. Figure 7 As shown, the data processing device 70 includes an acquisition module 72 and a first identification module 74. The data processing device 70 will be described below.

[0090] Acquisition module 72 is used to acquire multi-frame image data of the video;

[0091] The first recognition module 74 is connected to the acquisition module 72 and is used to identify the video scene of the video by inputting multi-frame image data into the video recognition model. The video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are related in time. The video recognition model is trained by multiple sets of data, which include multi-frame image data of the video and the video scene of the video.

[0092] It should be noted that the acquisition module 72 and the first identification module 74 mentioned above correspond to steps S202 to S204 in Embodiment 1. The instances and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0093] Example 3

[0094] According to an embodiment of the present invention, a data processing apparatus for implementing the second data processing method described above is also provided. Figure 8 This is a structural block diagram of the data processing device two according to Embodiment 3 of the present invention, as shown below. Figure 8 As shown, the data processing device 2 80 includes: a first display module 82, a receiving module 84, a second identification module 86, and a second display module 88. The data processing device 2 80 will be described below.

[0095] The first display module 82 is used to display video on the interactive interface;

[0096] The receiving module 84, connected to the first display module 82, is used to receive a request message on the interactive interface requesting to obtain a video scene.

[0097] The second recognition module 86 is connected to the receiving module 84 and is used to respond to the request message, acquire multi-frame image data of the video, and identify the video scene of the video by inputting the multi-frame image data into the video recognition model. The video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are related in time. The video recognition model is trained by multiple sets of data, which include multi-frame image data of the video and the video scene of the video.

[0098] The second display module 88 is connected to the second recognition module 86 and is used to display video scenes on the interactive interface.

[0099] It should be noted that the first display module 82, the receiving module 84, the second identification module 86, and the second display module 88 mentioned above correspond to steps S302 to S308 in Embodiment 1. The instances and application scenarios implemented by the above modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0100] Example 4

[0101] According to an embodiment of the present invention, a data processing apparatus for implementing the above-described data processing method three is also provided. Figure 9 This is a structural block diagram of the data processing device three according to Embodiment 4 of the present invention, as shown below. Figure 9 As shown, the data processing device 3 90 includes a live streaming module 92 and a third display module 94. The data processing device 3 90 will be described below.

[0102] Module 92 is used to broadcast live video on the live streaming interface;

[0103] The third display module 94 is connected to the aforementioned live streaming module 92 and is used to display the video scene of the video on the live streaming interface. The video scene of the video is obtained by recognizing the multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module, which is used to extract image features of the multi-frame image data that are related in time. The video recognition model is trained through multiple sets of data, which include: multi-frame image data of the video and the video scene of the video.

[0104] It should be noted that the aforementioned live streaming module 92 and the third display module 94 correspond to steps S402 to S404 in Embodiment 1. The instances and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0105] Example 5

[0106] According to an embodiment of the present invention, a data processing apparatus for implementing the above-described data processing method four is also provided. Figure 10 This is a structural block diagram of the data processing device four according to Embodiment 5 of the present invention, as follows: Figure 10 As shown, the data processing device 100 includes a playback module 102, a transmission module 104, and a fourth display module 106. The data processing device 100 will be described below.

[0107] Playback module 102 is used for playing videos on the front-end client;

[0108] The sending module 104 is connected to the playback module 102 and is used by the front-end client to send a request message for the video scene to be displayed to the back-end server.

[0109] The fourth display module 106, connected to the aforementioned sending module 104, is used by the front-end client to receive and display the video scene fed back by the back-end server. The video scene is obtained by the back-end server based on the multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module, which is used to extract image features of the multi-frame image data that are related in time. The video recognition model is trained through multiple sets of data, which include: multi-frame image data of the video and the video scene of the video.

[0110] It should be noted that the playback module 102, the sending module 104, and the fourth display module 106 correspond to steps S502 to S506 in Embodiment 1. The instances and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0111] Example 6

[0112] According to an embodiment of the present invention, a data processing apparatus for implementing the above-described data processing method five is also provided. Figure 11 This is a structural block diagram of the data processing apparatus five according to Embodiment 6 of the present invention, as follows: Figure 11 As shown, the data processing device 5110 includes: an acquisition module 112 and a training module 114. The data processing device 5110 will be described below.

[0113] The acquisition module 112 is used to acquire multiple sets of data, including: multiple frames of image data of the video and the video scene of the video; the training module 114 is connected to the acquisition module 112 and is used to perform machine learning training on the multiple sets of data by extracting image features of the multiple frames of image data that are related in time from the multiple sets of data, and to obtain a video recognition model.

[0114] It should be noted that the acquisition module 112 and training module 114 mentioned above correspond to steps S602 to S604 in Embodiment 1. The instances and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0115] Example 7

[0116] Embodiments of the present invention can provide a computer terminal, which can be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0117] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0118] In this embodiment, the computer terminal described above can execute the program code for the following steps in the data processing method of the application: acquiring multi-frame image data of a video; identifying the video scene of the video by inputting the multi-frame image data into a video recognition model, wherein the video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are related in time sequence, wherein the video recognition model is trained by multiple sets of data, and the multiple sets of data include: multi-frame image data of the video and the video scene of the video.

[0119] Optionally, Figure 12 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Figure 12 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 122 and memory 124.

[0120] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the aforementioned data processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: acquiring multi-frame image data of a video; identifying the video scene of the video by inputting the multi-frame image data into a video recognition model, wherein the video recognition model includes a video feature module, which is used to extract image features of the multi-frame image data that are correlated in time; wherein the video recognition model is trained by multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video.

[0122] Optionally, the processor may also execute program code for the following steps: The video feature module includes a global feature submodule and an object local feature submodule. By inputting multi-frame image data into the video recognition model, the video scene of the video is identified by: extracting the video global features of each frame of the multi-frame image data through the global feature submodule, wherein the video global features include features of each frame of the image data exceeding a predetermined dimension, as well as temporal information between the multi-frame image data; extracting the object local features of each frame of the multi-frame image data through the object local feature submodule; and identifying the video scene of the video based on the video global features and the object local features.

[0123] Optionally, the processor may also execute program code with the following steps: The video recognition model further includes an attention module, wherein recognizing the video scene based on global video features and local object features includes: assigning a first weight coefficient to the local object features through the attention module based on the global video features, and assigning a second weight coefficient to the global video features of each frame of image data; updating the local object features based on the first weight coefficient to obtain updated local object features, and updating the global video features of each frame of image data based on the second weight coefficient to obtain updated global video features of each frame of image data; and recognizing the video scene based on the updated local object features and the updated global video features of each frame of image data.

[0124] Optionally, the processor may also execute program code for the following steps: The video recognition model further includes a classifier, wherein identifying the video scene of the video based on the obtained updated local features of the object and the updated global features of the video for each frame of image data includes: processing the updated local features of the object and the updated global features of the video for each frame of image data through an attention module to obtain video output features; processing the video output features with a classifier to obtain classification results, and determining the video scene of the video based on the classification results.

[0125] Optionally, the processor may also execute program code for the following steps: The video recognition model further includes a classifier, wherein identifying the video scene of the video based on the obtained updated local features of the object and the updated global features of each frame of image data includes: using the classifier to classify the updated local features of the object and the updated global features of each frame of image data, respectively obtaining classification results corresponding to the updated local features of the object and classification results corresponding to the updated global features of each frame of image data, wherein each classification result corresponds to a video scene; determining the video scene of the video based on the classification results corresponding to the updated local features of the object and the classification results corresponding to the updated global features of each frame of image data.

[0126] Optionally, the processor may also execute program code that performs the following steps: acquiring multi-frame image data of a video by at least one of the following methods: acquiring consecutive multi-frame image data of a video; sampling the video multiple times according to a fixed frame interval to obtain multi-frame image data of the video.

[0127] Optionally, the processor may also execute program code that includes at least one of the following: live video and recorded video.

[0128] Optionally, the processor may also execute program code that performs the following steps: the multi-frame image data is 16 frames of image data.

[0129] The processor can access information and applications stored in memory via a transmission device to perform the following steps: displaying a video on an interactive interface; receiving a request message on the interactive interface requesting the acquisition of a video scene; responding to the request message, acquiring multi-frame image data of the video, and identifying the video scene of the video by inputting the multi-frame image data into a video recognition model, wherein the video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are temporally correlated, wherein the video recognition model is trained through multiple sets of data, the multiple sets of data including: multi-frame image data of the video and the video scene of the video; and displaying the video scene on the interactive interface.

[0130] The processor can access information and applications stored in memory via a transmission device to perform the following steps: live streaming video on the live streaming interface; displaying the video scene on the live streaming interface, wherein the video scene is obtained by recognizing multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are correlated in time. The video recognition model is trained using multiple sets of data, which include multi-frame image data of the video and the video scene of the video.

[0131] The processor can access information and applications stored in memory via a transmission device to perform the following steps: the front-end client plays a video; the front-end client sends a request message for displaying a video scene to the back-end server; the front-end client receives the video scene from the back-end server and displays it. The video scene is obtained by the back-end server using a video recognition model based on multi-frame image data of the video. The video recognition model includes a video feature module, which extracts image features from time-related multi-frame image data. The video recognition model is trained using multiple sets of data, which include multi-frame image data of the video and the video scene of the video.

[0132] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: acquiring multiple sets of data, wherein the multiple sets of data include: multiple frames of image data of a video and the video scene of the video; extracting image features of the time-related multiple frames of image data from the multiple sets of data respectively, performing machine learning training on the multiple sets of data to obtain a video recognition model.

[0133] This invention provides a data processing method. By acquiring multi-frame image data from a video and inputting this data into a video recognition model that includes a video feature module, the method extracts image features from the temporally related multi-frame image data. This achieves the goal of identifying video scenes, thus realizing the technical effect of identifying video scene types based on video content and solving the technical problem of inaccurately identifying video scenes based on video content.

[0134] Those skilled in the art will understand that Figure 12 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 12 This does not limit the structure of the aforementioned electronic devices. For example, a computer terminal may also include components that are more... Figure 12 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 12 The different configurations shown.

[0135] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0136] Example 8

[0137] Embodiments of the present invention also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in Embodiment 1.

[0138] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0139] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring multi-frame image data of a video; identifying the video scene of the video by inputting the multi-frame image data into a video recognition model, wherein the video recognition model includes a video feature module, which is used to extract image features of the multi-frame image data that are temporally related, wherein the video recognition model is trained by multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video.

[0140] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: The video feature module includes a global feature submodule and an object local feature submodule. Identifying the video scene by inputting multi-frame image data into the video recognition model includes: extracting global video features from each frame of the multi-frame image data using the global feature submodule, wherein the global video features include features exceeding a predetermined dimension for each frame of the image data, and temporal information between the multi-frame image data; extracting object local features from each frame of the multi-frame image data using the object local feature submodule; and identifying the video scene based on the global video features and the object local features.

[0141] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the video recognition model further includes an attention module, wherein identifying the video scene based on video global features and object local features includes: assigning a first weight coefficient to the object local features through the attention module according to the video global features, and assigning a second weight coefficient to the video global features of each frame of image data respectively; updating the object local features according to the first weight coefficient to obtain updated object local features, and updating the video global features of each frame of image data according to the second weight coefficient to obtain updated video global features of each frame of image data; and identifying the video scene of the video based on the obtained updated object local features and the updated video global features of each frame of image data.

[0142] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the video recognition model further includes a classifier, wherein identifying the video scene of the video based on the obtained updated local features of the object and the updated global features of the video for each frame of image data includes: processing the updated local features of the object and the updated global features of the video for each frame of image data through an attention module to obtain video output features; processing the video output features with a classifier to obtain a classification result, and determining the video scene of the video based on the classification result.

[0143] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the video recognition model further includes a classifier, wherein identifying the video scene of the video based on the video output features includes: using the classifier to classify the updated local features of the object and the updated global video features of each frame of image data, respectively obtaining a classification result corresponding to the updated local features of the object and a classification result corresponding to the updated global video features of each frame of image data, wherein each classification result corresponds to a video scene; determining the video scene of the video based on the classification result corresponding to the updated local features of the object and the classification result corresponding to the updated global video features of each frame of image data.

[0144] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring multi-frame image data of a video by at least one of the following methods: acquiring consecutive multi-frame image data in the video; sampling the video multiple times according to a fixed frame interval to obtain multi-frame image data in the video.

[0145] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the video includes at least one of the following: live video, recorded video.

[0146] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the multi-frame image data is 16 frames of image data.

[0147] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: displaying a video on an interactive interface; receiving a request message on the interactive interface requesting the acquisition of a video scene; responding to the request message, acquiring multi-frame image data of the video, and identifying the video scene of the video by inputting the multi-frame image data into a video recognition model, wherein the video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are temporally related, wherein the video recognition model is trained using multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video; and displaying the video scene on the interactive interface.

[0148] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: broadcasting video live on the live streaming interface; displaying the video scene of the video on the live streaming interface, wherein the video scene of the video is obtained by recognizing the multi-frame image data of the video using a video recognition model, the video recognition model including a video feature module, the video feature module being used to extract image features of the multi-frame image data that are temporally related, wherein the video recognition model is trained through multiple sets of data, the multiple sets of data respectively including: multi-frame image data of the video and the video scene of the video.

[0149] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the front-end client plays a video; the front-end client sends a request message for requesting the display of a video scene to the back-end server; the front-end client receives the video scene from the back-end server and displays the video scene, wherein the video scene is obtained by the back-end server based on the multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module, which is used to extract image features of multi-frame image data that are correlated in time. The video recognition model is trained using multiple sets of data, which respectively include: multi-frame image data of the video and the video scene of the video.

[0150] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: collecting multiple sets of data, wherein the multiple sets of data respectively include: multi-frame image data of a video and the video scene of the video; by extracting image features of the multi-frame image data that are temporally related in the multiple sets of data respectively, performing machine learning training on the multiple sets of data to obtain a video recognition model.

[0151] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0152] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0157] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: Acquire multi-frame image data from a video; By inputting the multi-frame image data into a video recognition model, the video scene of the video is identified. The video recognition model includes a video feature module and an attention module. The video feature module is used to extract image features of the multi-frame image data that are temporally related. The video recognition model is trained through multiple sets of data, which respectively include: multi-frame image data of the video and the video scene of the video. The image features include global video features and local object features. The method further includes: preprocessing the global features of the video, the local features of the object, and the classification node to obtain a feature sequence; inputting the feature sequence into the attention module to obtain video output features, wherein the video output features are used to represent the features corresponding to the classification node to determine the video scene of the video.

2. The method according to claim 1, characterized in that, The video feature module includes a global feature submodule and an object local feature submodule. By inputting the multi-frame image data into the video recognition model, the video scene of the video is identified, including: The global feature submodule extracts video global features from each frame of the multi-frame image data, wherein the video global features include features of each frame of image data exceeding a predetermined dimension, as well as temporal information between the multi-frame image data. The object local feature submodule extracts the object local features of each frame of image data in the multi-frame image data; Based on the global features of the video and the local features of the object, the video scene of the video is identified.

3. The method according to claim 2, characterized in that, Based on the global features of the video and the local features of the object, the video scene identified in the video includes: Based on the video global features, the attention module assigns a first weight coefficient to the local features of the object and assigns a second weight coefficient to the video global features of each frame of image data. The local features of the object are updated according to the first weight coefficient to obtain the updated local features of the object, and the video global features of each frame of image data are updated according to the second weight coefficient to obtain the updated video global features of each frame of image data. Based on the updated local features of the object and the updated global video features of each frame of image data, the video scene of the video is identified.

4. The method according to claim 3, characterized in that, The video recognition model further includes a classifier, wherein, based on the updated local features of the object and the updated global video features of each frame of image data, the video scene identified includes: The attention module processes the updated local features of the object and the updated global video features of each frame of image data to obtain the video output features; The video output features are processed using the classifier to obtain a classification result, and the video scene of the video is determined based on the classification result.

5. The method according to claim 3, characterized in that, The video recognition model further includes a classifier, wherein, based on the updated local features of the object and the updated global video features of each frame of image data, the video scene identified includes: The classifier is used to classify the updated local features of the object and the updated global video features of each frame of image data, respectively obtaining the classification result corresponding to the updated local features of the object and the classification result corresponding to the updated global video features of each frame of image data, wherein each classification result corresponds to a video scene; The video scene is determined based on the classification results corresponding to the updated local features of the object and the classification results corresponding to the video global features of each frame of image data.

6. The method according to any one of claims 1 to 5, characterized in that, The video multi-frame image data is obtained by at least one of the following methods: Acquire consecutive multi-frame image data from the video; The video is sampled multiple times at fixed frame intervals to obtain multi-frame image data in the video.

7. The method according to claim 6, characterized in that, The video includes at least one of the following: live video, recorded video.

8. The method according to claim 7, characterized in that, The multi-frame image data consists of 16 frames of image data.

9. A data processing method, characterized in that, include: Display the video on the interactive interface; A request message is received on the interactive interface requesting the acquisition of the video scene of the video; In response to the request message, the system acquires multi-frame image data of the video and identifies the video scene of the video by inputting the multi-frame image data into a video recognition model. The video recognition model includes a video feature module and an attention module. The video feature module is used to extract image features of the multi-frame image data that are temporally related. The video recognition model is trained using multiple sets of data, which respectively include: multi-frame image data of the video and the video scene of the video. The image features include global video features and local object features. The video scene is displayed on the interactive interface; The method further includes: preprocessing the global features of the video, the local features of the object, and the classification node to obtain a feature sequence; inputting the feature sequence into the attention module to obtain video output features, wherein the video output features are used to represent the features corresponding to the classification node to determine the video scene of the video.

10. A data processing method, characterized in that, include: Live video stream on the live streaming interface; The live streaming interface displays the video scene of the video, which is obtained by recognizing the video scene based on the multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module and an attention module. The video feature module is used to extract image features of the multi-frame image data that are temporally related. The video recognition model is trained through multiple sets of data, which respectively include: multi-frame image data of the video and the video scene of the video. The image features include global video features and local object features. The method further includes: preprocessing the global features of the video, the local features of the object, and the classification node to obtain a feature sequence; inputting the feature sequence into the attention module to obtain video output features, wherein the video output features are used to represent the features corresponding to the classification node to determine the video scene of the video.

11. A data processing method, characterized in that, include: The front-end client plays the video; The front-end client sends a request message to the back-end server to request the display of the video scene. The front-end client receives and displays the video scene fed back by the back-end server. The video scene is obtained by the back-end server based on the multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module and an attention module. The video feature module is used to extract image features of the multi-frame image data that are temporally related. The video recognition model is trained through multiple sets of data, which respectively include: multi-frame image data of the video and the video scene of the video. The image features include global video features and local object features. The method further includes: preprocessing the global features of the video, the local features of the object, and the classification node to obtain a feature sequence; inputting the feature sequence into the attention module to obtain video output features, wherein the video output features are used to represent the features corresponding to the classification node to determine the video scene of the video.

12. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire multi-frame image data from the video. The first recognition module is used to identify the video scene of the video by inputting the multi-frame image data into a video recognition model. The video recognition model includes a video feature module and an attention module. The video feature module is used to extract image features of the multi-frame image data that are temporally related. The video recognition model is trained by multiple sets of data, which respectively include: multi-frame image data of the video and the video scene of the video. The image features include global video features and local object features. The device further includes: a processing module, used to preprocess the global features of the video, the local features of the object, and the classification node to obtain a feature sequence; and input the feature sequence into the attention module to obtain video output features, wherein the video output features are used to represent the features corresponding to the classification node to determine the video scene of the video.

13. A data processing apparatus, characterized in that, include: The first display module is used to display video on the interactive interface; A receiving module is used to receive a request message on the interactive interface requesting the acquisition of the video scene of the video; The second recognition module is used to respond to the request message, acquire multi-frame image data of the video, and identify the video scene of the video by inputting the multi-frame image data into a video recognition model. The video recognition model includes a video feature module and an attention module. The video feature module is used to extract image features of the multi-frame image data that are temporally related. The video recognition model is trained by multiple sets of data, which respectively include: multi-frame image data of the video and the video scene of the video. The image features include global video features and local object features. The second display module is used to display the video scene on the interactive interface; The device further includes: a processing module, used to preprocess the global features of the video, the local features of the object, and the classification node to obtain a feature sequence; and input the feature sequence into the attention module to obtain video output features, wherein the video output features are used to represent the features corresponding to the classification node to determine the video scene of the video.

14. A data processing apparatus, characterized in that, include: The live streaming module is used to broadcast videos on the live streaming interface; The third display module is used to display the video scene of the video on the live streaming interface. The video scene of the video is obtained by recognizing the multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module and an attention module. The video feature module is used to extract image features of the multi-frame image data that are temporally related. The video recognition model is trained through multiple sets of data, which respectively include: multi-frame image data of the video and the video scene of the video. The image features include global video features and local object features. The device further includes: a processing module, used to preprocess the global features of the video, the local features of the object, and the classification node to obtain a feature sequence; and input the feature sequence into the attention module to obtain video output features, wherein the video output features are used to represent the features corresponding to the classification node to determine the video scene of the video.

15. A data processing apparatus, characterized in that, include: The playback module is used for playing videos on the front-end client. The sending module is used by the front-end client to send a request message for the video scene to be displayed to the back-end server; The fourth display module is used for the front-end client to receive and display the video scene fed back by the back-end server. The video scene is obtained by the back-end server based on the multi-frame image data of the video using a video recognition model. The video recognition model includes a video feature module and an attention module. The video feature module is used to extract image features of the multi-frame image data that are temporally related. The video recognition model is trained through multiple sets of data, which respectively include: multi-frame image data of the video and the video scene of the video. The image features include global video features and local object features. The device further includes: a processing module, used to preprocess the global features of the video, the local features of the object, and the classification node to obtain a feature sequence; and input the feature sequence into the attention module to obtain video output features, wherein the video output features are used to represent the features corresponding to the classification node to determine the video scene of the video.

16. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the data processing method according to any one of claims 1 to 11.

17. A computer device, characterized in that, include: Memory and processor The memory stores computer programs; The processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, the processor performs the data processing method according to any one of claims 1 to 11.

18. A data processing method, characterized in that, include: Multiple sets of data are collected, wherein the multiple sets of data include: multi-frame image data of the video and the video scene of the video; By extracting image features from multiple frames of image data that are temporally related in the multiple sets of data, and then performing machine learning training on the multiple sets of data, a video recognition model is obtained.

19. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire multiple sets of data, wherein the multiple sets of data include: multi-frame image data of the video and the video scene of the video; The training module is used to perform machine learning training on the multiple sets of data by extracting image features from the multiple frames of image data that are temporally related, and to obtain a video recognition model.

Images