Data processing method, data processing device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method, a data processing device, electronic equipment, a storage medium and a program product, belongs to the technical field of computers, and particularly relates to an artificial intelligence technology. The implementation scheme comprises: receiving an input video; for each frame in the input video, determining an image, a text and an audio corresponding to the frame; performing format conversion on at least one of the image, the text and the audio corresponding to the frame to obtain corresponding image representation, text representation and audio representation; splicing the image representation, the text representation and the audio representation corresponding to the frame to obtain the content representation of the frame; encoding the content representation to obtain an encoding feature of the frame; and determining the coding characteristics of the input video according to the coding characteristics of the plurality of frames included in the input video.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to artificial intelligence technology, and specifically to a data processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Today, the field of natural language is moving towards the era of ultra-large-scale models. Through super computing power, training ultra-large parameter models on massive text data can make the output language model have the general semantic understanding and generation capabilities of multi-task and few-sample learning. In order to develop more general large models, in addition to text input, today's large models are developing in the direction of multi-modality (pictures, audio and video, etc.).

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0006] According to one aspect of the present disclosure, a data processing method is provided, comprising: receiving an input video; for each frame in the input video, determining an image, text, and audio corresponding to the frame; performing format conversion on at least one of the image, text, and audio corresponding to the frame to obtain a corresponding image representation, text representation, and audio representation; splicing the image representation, text representation, and audio representation corresponding to the frame to obtain a content representation of the frame; encoding the content representation to obtain a coding feature of the frame; and determining the coding features of the input video based on the coding features of multiple frames included in the input video.

[0007] According to another aspect of the present disclosure, a data processing device is provided, including a receiving unit configured to receive an input video; an encoding unit configured to: for each frame in the input video, determine the image, text and audio corresponding to the frame; perform format conversion on at least one of the image, text and audio corresponding to the frame to obtain a corresponding image representation, text representation and audio representation; splice the image representation, text representation and audio representation to obtain a content representation of the frame; encode the content representation to obtain an encoding feature of the frame; and a feature determination unit configured to determine the encoding features of the input video based on the encoding features of multiple frames included in the input video.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method according to an embodiment of the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute a method according to an embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method according to the embodiment of the present disclosure when being executed by a processor.

[0011] According to one or more embodiments of the present disclosure, unified encoding of video data may be implemented.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0015] Figure 2An exemplary flow chart of a data processing method according to an embodiment of the present disclosure is shown;

[0016] Figure 3 The principle of the data processing process according to the embodiment of the present disclosure is shown;

[0017] Figure 4 An exemplary block diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0018] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0021] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0022] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0024] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the data processing method according to an embodiment of the present disclosure.

[0025] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0026] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0027] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to obtain digital input data and output data processing results. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0028] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0029] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0030] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0031] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0032] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0033] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0034] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0035] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0036] Figure 1The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0037] Video is a common multimodal data. Video contains information such as images, text, and audio aligned in the time dimension, which makes the encoding of video modality data modeling a complex task.

[0038] The method adopted in the related art is to split the video into multiple modalities, and then model the data of each modality separately. For example, the Video-ChatGPT model is used to model the picture modality in the video. However, this modeling method will cause the loss of text and audio information in the video, making it impossible to restore the content of the video itself. You can also use the VideoLLaMA series of models, use the visual encoder and audio encoder to model the picture and audio separately, and then merge the modeled outputs by splicing. However, using different modal encoders to model each modality in the video will lead to an inconsistent and complicated modeling process, and will not utilize the characteristics of video time dimension alignment. The subsequent modal splicing operation will also lead to low modeling efficiency.

[0039] In order to provide a better video modeling method for model processing, the present disclosure provides a new data processing method.

[0040] Figure 2 An exemplary flow chart of a data processing method according to an embodiment of the present disclosure is shown.

[0041] In step S202, an input video is received.

[0042] In step S204, each frame in the input video is encoded. Step S204 includes: determining the image, text and audio corresponding to the frame; performing format conversion on at least one of the image, text and audio corresponding to the frame to obtain a corresponding image representation, text representation and audio representation; splicing the image representation, text representation and audio representation to obtain a content representation of the frame; and encoding the content representation to obtain a coding feature of the frame.

[0043] In step S206, the encoding features of the input video are determined according to the encoding features of a plurality of frames included in the input video.

[0044] By using the data processing method provided by the embodiments of the present disclosure, unified encoding of video data can be achieved, and the characteristic of the video itself containing multi-modal aligned input is taken into account during the encoding process, thereby completing efficient unified video modeling and reducing information loss in the video model.

[0045] The principles of the present disclosure will be described in detail below.

[0046] In step S202, an input video may be received. The video includes information of at least two modes including images, text, and audio. For example, the video includes multiple image frames and audio played together with the multiple image frames, including background audio and narration audio, etc. In some cases, the video also includes subtitles played together with the background audio. The text included in the video may include text in the subtitles, and may also include text recognized from the narration audio. It is understandable that when humans watch the video, the above-mentioned images, texts, and audios are played in a time-aligned manner, and the contents of the images, texts, and audios echo each other, thereby providing users with multi-faceted information.

[0047] However, in the related art, when a machine learning model is needed to process video data, since information of different modalities has different representation forms, in order to encode the video data, the information of different modalities included in the video needs to be encoded separately, and then spliced ​​together and input into the machine learning model. However, in this process, the cross-modal information existing in the multimodal information in the video under the condition of time alignment cannot be reflected.

[0048] In step S204, each frame in the input video is encoded. Step S204 includes: determining the image, text and audio corresponding to the frame; performing format conversion on at least one of the image, text and audio corresponding to the frame to obtain a corresponding image representation, text representation and audio representation; splicing the image representation, text representation and audio representation to obtain a content representation of the frame; and encoding the content representation to obtain a coding feature of the frame.

[0049] In step S204, the relevant multimodal information is encoded in units of one frame. The encoded features of each frame include the content of the multimodal information that is time-aligned with the frame. Therefore, it is possible to encode the image information and the audio and text information related to the image information together. When the encoded features of the video are processed using a machine learning model, the model can understand the information of the time alignment of the multimodal information while understanding the information of the image, audio, and text itself.

[0050] Wherein, determining the image, text and audio corresponding to the frame includes: determining the video image displayed at the frame moment; determining the text information in the video within a predetermined time range including the frame moment; and determining the audio information in the video within a predetermined time range including the frame moment. Wherein, the time length of the predetermined time range for determining the text information and the audio information can be determined based on the time length between two adjacent frames. In the example, the predetermined time range can be the time length between two adjacent frames. According to actual conditions, the predetermined time range can also be set to other lengths.

[0051] In some embodiments, the image representation, text representation, and audio representation corresponding to each frame may have the same format.

[0052] In the example, the image representation, text representation and audio representation corresponding to each frame are all in picture format. In this case, the image representation can be the image itself corresponding to the frame of video. For example, format conversion of at least one of the image, text and audio corresponding to the frame may include: generating a text image including text according to the text content; extracting the spectrum of the audio to obtain a spectrogram corresponding to the audio. Any text processing method or image processing method can be used to generate a text image including text, such as generating a page including text content and taking a screenshot of the page. For example, the audio information can be processed using a Mel filter to obtain at least one Mel spectrogram corresponding to the audio information. Among them, the text image and the spectrogram can have the same size as the image of each frame. The content representation of the frame can be obtained by splicing the image, text image and spectrogram corresponding to each frame. Thus, the content representation of each frame can include the content of multimodal information in a unified format and time-aligned. Information of different modes can be spliced ​​on any dimension of the content representation according to actual conditions.

[0053] In other examples, the image representation, text representation, and audio representation corresponding to each frame may be unified into other formats other than images, such as text form, audio form, or vector form, tensor form, etc. of any mapping space. Those skilled in the art may determine the unified format and format conversion method of multimodal information according to actual conditions.

[0054] In the case where the content representation of each frame includes multimodal information in a unified format, a single encoder can be used to encode the content representation. Taking the case where the content representation of each frame is in a picture format, a visual encoder can be used to encode the content representation, thereby achieving the effect of unified modeling of time-aligned multimodal information. Using this method, there is no need to train different encoders for information in different modalities, thereby simplifying the video encoding process and reducing the information loss and operational complexity caused by separate modal modeling.

[0055] In step S206, the encoding features of the input video are determined according to the encoding features of a plurality of frames included in the input video.

[0056] The coding features of the input video can be processed using a machine learning model to obtain a prediction result corresponding to the input video. For example, the above-mentioned machine learning model can be a large language model. An exemplary application scenario may include that a user inputs a video and enters a prompt message "Please summarize the video content". The input video can be encoded using the method in step S204, and the prompt message and the coding features of the input video are input into the large language model together, and the prediction result output by the large language model is obtained, that is, the large language model summarizes the video content.

[0057] It is understandable that, according to actual conditions, those skilled in the art may also input the coding features of the input video into other types of machine learning models to obtain the output results required in other application scenarios. In order to enable the machine learning model to process the coding features of the input video, the coding features may also be transformed in any possible way, thereby achieving the effect of obtaining video information using various different machine learning models.

[0058] In the data processing method provided in the embodiments of the present disclosure, modeling is performed for the multimodal information of images, texts, and audios in the video. Compared with modeling only a single modality, information loss is reduced, so the restoration quality of the information of each modality of the video in subsequent operations can be improved. In addition, by using a unified encoder to model multimodal data, the alignment cost and the overhead of multiple calls are reduced compared to modeling different modalities by calling their own encoders. And this modeling method also makes the information fusion between modalities more unified.

[0059] Figure 3 The principle of the data processing process according to the embodiment of the present disclosure is shown.

[0060] Figure 3 3 shows an image representation 301, a text representation 302, and an audio representation 303 of the t-th frame and the t+1-th frame of the input video. The image representation 301, the text representation 302, and the audio representation 303 are all in picture format. The image representation, the text representation, and the audio representation of the t-th frame can be spliced ​​to obtain a content representation 304 of the t-th frame and a content representation 305 of the t+1-th frame.

[0061] The visual encoder can be used to encode the content representation 304 of the tth frame and the content representation 305 of the t+1th frame. In a similar manner, the content representations of the 1st to nth frames (n is a positive integer) of the input video can be encoded to obtain the encoding features 306 of the input video. Figure 3 The numbers shown in the coding feature 306 shown in FIG. 3 correspond to the index of the frame number.

[0062] The large language model 320 may be used to process the consensus encoding features 306 of the input video to obtain a prediction result corresponding to the input video.

[0063] Figure 4 An exemplary block diagram of a data processing device according to an embodiment of the present disclosure is shown. Figure 4 The data processing apparatus shown in FIG. 1 is used to perform the combined Figure 2 A data processing method 200 is described.

[0064] like Figure 4 As shown, the data processing device 400 includes a receiving unit 410, an encoding unit 420 and a feature determination unit 430.

[0065] The receiving unit 410 may be configured to receive an input video.

[0066] The encoding unit 420 can be configured to: for each frame in the input video, determine the image, text and audio corresponding to the frame; perform format conversion on at least one of the image, text and audio corresponding to the frame to obtain the corresponding image representation, text representation and audio representation; splice the image representation, text representation and audio representation to obtain the content representation of the frame; encode the content representation to obtain the encoding features of the frame.

[0067] The feature determination unit 430 may be configured to determine the encoding feature of the input video according to encoding features of a plurality of frames included in the input video.

[0068] By using the data processing device provided by the embodiments of the present disclosure, unified encoding of video data can be achieved, and the characteristic of the video itself containing multi-modal aligned input is taken into account during the encoding process, thereby completing efficient unified video modeling and reducing information loss in the video model.

[0069] In some embodiments, for each frame in the input video, determining the image, text, and audio corresponding to the frame includes: determining the video image displayed at the frame moment; determining the text information in the video within a predetermined time range including the frame moment; and determining the audio information in the video within a predetermined time range including the frame moment.

[0070] In some embodiments, the image representation, the text representation, and the audio representation are all in picture format.

[0071] In some embodiments, format conversion of at least one of the image, text, and audio corresponding to the frame includes: generating a text picture including text according to the text content; and extracting the spectrum of the audio to obtain a spectrum diagram corresponding to the audio.

[0072] In some embodiments, the image, text picture, and spectrogram have the same size.

[0073] In some embodiments, the data processing device 400 further includes a prediction unit, which is configured to: process the encoding features of the input video using the large language model to obtain a prediction result corresponding to the input video.

[0074] It should be understood that Figure 4 The various modules or units of the apparatus 400 shown in FIG. 4 can be used in conjunction with the reference Figure 2 The above-described method 200 corresponds to each step in the method 200. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 400 and the modules and units included therein. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 400 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0075] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various units discussed herein may be separated into multiple units, and / or at least some functionality of multiple units may be combined into a single unit.

[0076] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 4 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 410 to 430 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0077] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0078] According to another aspect of the present disclosure, an electronic device is also provided, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method according to an embodiment of the present disclosure.

[0079] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, where the computer instructions are used to enable the computer to execute the data processing method according to the embodiment of the present disclosure.

[0080] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the data processing method according to the embodiment of the present disclosure when executed by a processor.

[0081] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0082] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0083] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device that can input information to the electronic device 500. The input unit 506 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 507 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 can include but is not limited to a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0084] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the method 200 in any other appropriate manner (e.g., by means of firmware).

[0085] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0086] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0087] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0089] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0090] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0091] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0092] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A data processing method, comprising: Receive input video; For each frame in the input video, Determine the image, text and audio corresponding to the frame; Performing format conversion on at least one of the image, text, and audio corresponding to the frame to obtain corresponding image representation, text representation, and audio representation; splicing the image representation, text representation and audio representation corresponding to the frame to obtain a content representation of the frame; Encoding the content representation to obtain an encoding feature of the frame; The encoding feature of the input video is determined according to encoding features of a plurality of frames included in the input video.

2. The data processing method according to claim 1, wherein: For each frame in the input video, determining the image, text, and audio corresponding to the frame includes: Determine the video image displayed at the frame time; Determining text information in the video within a predetermined time range including the frame moment; and The audio information in the video within a predetermined time range including the frame time is determined.

3. The data processing method according to claim 1, wherein the image representation, text representation and audio representation are all in picture format.

4. The data processing method according to claim 3, wherein: Converting the format of at least one of the image, text, and audio corresponding to the frame includes: Generating a text image including the text according to the text content; The frequency spectrum of the audio is extracted to obtain a spectrogram corresponding to the audio.

5. The data processing method according to claim 4, wherein: The image, the text picture and the spectrogram have the same size.

6. The data processing method according to any one of claims 1 to 5, further comprising: The encoding features of the input video are processed using a large language model to obtain a prediction result corresponding to the input video.

7. A data processing device, comprising: A receiving unit configured to receive an input video; The encoding unit is configured as follows: For each frame in the input video, Determine the image, text and audio corresponding to the frame; Performing format conversion on at least one of the image, text, and audio corresponding to the frame to obtain corresponding image representation, text representation, and audio representation; splicing the image representation, text representation and audio representation to obtain a content representation of the frame; Encoding the content representation to obtain an encoding feature of the frame; The feature determination unit is configured to determine the encoding feature of the input video according to the encoding features of a plurality of frames included in the input video.

8. The data processing apparatus according to claim 7, wherein: For each frame in the input video, determining the image, text, and audio corresponding to the frame includes: Determine the video image displayed at the frame time; Determining text information in the video within a predetermined time range including the frame moment; and The audio information in the video within a predetermined time range including the frame time is determined.

9. The data processing apparatus of claim 7, wherein the image representation, text representation and audio representation are all in picture format.

10. The data processing apparatus according to claim 9, wherein: Converting the format of at least one of the image, text, and audio corresponding to the frame includes: Generating a text image including the text according to the text content; The frequency spectrum of the audio is extracted to obtain a spectrogram corresponding to the audio.

11. The data processing apparatus according to claim 10, wherein: The image, the text picture and the spectrogram have the same size.

12. The data processing device as described in any one of claims 7-11 further includes a prediction unit, wherein the prediction unit is configured to: use a large language model to process the encoding features of the input video to obtain a prediction result corresponding to the input video.

13. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images