Feature data compression method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a feature data compression method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an initial feature sequence of a target video, and enabling the initial feature sequence to represent the video content features of the target video; respectively generating enhanced query data and formatted feature data according to the initial feature sequence; respectively processing the enhanced query data and the formatted feature data through a feature enhancement model to obtain compressed feature data of the target video, the feature enhancement model including a plurality of cross attention layers, and the cross attention layers including a plurality of cross attention layers; the plurality of cross attention layers are used for gradually extracting key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate the compressed feature data, so that the problem that the data volume of the feature data of the to-be-processed video (target video) is too large is solved, and the processing efficiency is improved. And thus, the processing efficiency and accuracy of the multi-modal large model processing data are improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology, and in particular, to a method, device, electronic device and storage medium for compressing feature data. Background Art

[0002] At present, with the application and development of large language models in multiple fields, a variety of multimodal large models have been derived. Especially in the application field of video content analysis and understanding, multimodal large models with visual perception capabilities are widely used by users.

[0003] In the prior art, when users use a large multimodal model to process video content, they often encode the video content of the video to be processed to generate feature data of the video. Then, the large language model couples the feature data with the user command features input by the user to generate video information corresponding to the user command features.

[0004] The above solutions in the prior art have the problem that the amount of feature data of the video to be processed is too large. Summary of the invention

[0005] The embodiments of the present disclosure provide a method, device, electronic device and storage medium for compressing feature data to overcome the problem of excessive amount of feature data of a video to be processed.

[0006] In a first aspect, an embodiment of the present disclosure provides a feature data compression method, comprising:

[0007] An initial feature sequence of a target video is obtained, wherein the initial feature sequence characterizes the video content features of the target video; based on the initial feature sequence, enhanced query data and formatted feature data are generated respectively, wherein the enhanced query data is constructed based on global feature data, the global feature data is data after dimension reduction is performed on the time dimension of the initial feature sequence, and the formatted feature data is data after the initial feature sequence is reformatted based on the target data format; the enhanced query data and the formatted feature data are processed respectively through a feature enhancement model to obtain compressed feature data of the target video, wherein the feature enhancement model includes a plurality of cross-attention layers, and the plurality of cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate the compressed feature data.

[0008] In a second aspect, an embodiment of the present disclosure provides a feature data compression device, including:

[0009] An acquisition module, used to acquire an initial feature sequence of a target video, wherein the initial feature sequence represents video content features of the target video;

[0010] A generating module, configured to generate enhanced query data and formatted feature data according to the initial feature sequence, wherein the enhanced query data is constructed based on global feature data, the global feature data is data after dimension reduction is performed on the time dimension of the initial feature sequence, and the formatted feature data is data after the initial feature sequence is formatted based on a target data format;

[0011] A processing module is used to process the enhanced query data and the formatted feature data respectively through a feature enhancement model to obtain compressed feature data of the target video, wherein the feature enhancement model includes a plurality of cross-attention layers, and the plurality of cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate the compressed feature data.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0013] The memory stores computer-executable instructions;

[0014] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the feature data compression method described in the first aspect and various possible designs of the first aspect.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the characteristic data compression method described in the first aspect and various possible designs of the first aspect is implemented.

[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the characteristic data compression method described in the first aspect and various possible designs of the first aspect.

[0017] The feature data compression method, device, electronic device and storage medium provided in the present embodiment obtain an initial feature sequence of a target video, wherein the initial feature sequence characterizes the video content features of the target video; based on the initial feature sequence, enhanced query data and formatted feature data are generated respectively, wherein the enhanced query data is constructed based on global feature data, wherein the global feature data is data after dimension reduction is performed on the time dimension of the initial feature sequence, and the formatted feature data is data after the initial feature sequence is reformatted based on the target data format; the enhanced query data and the formatted feature data are processed respectively through a feature enhancement model to obtain compressed feature data of the target video, wherein the feature enhancement model includes a plurality of cross-attention layers, and the plurality of cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate the compressed feature data. By utilizing the initial feature sequence, enhanced query data constructed by global feature data is generated, and then the enhanced query data and the formatted feature data generated after formatting the initial feature sequence are processed through a cross-attention layer to obtain compressed feature data. Since the enhanced query data contains global feature data that is reduced in dimension based on the time dimension, video features are gradually extracted and compressed based on it through multiple layers of cross-attention layers to achieve continuous attention to key features. On the one hand, the data volume of key features extracted through cross-attention is reduced. On the other hand, the information in the global feature data is utilized to improve the accuracy of the obtained key features, and the feature validity of the compressed feature data finally output is improved, which solves the problem of excessive data volume of feature data of the video to be processed (target video), thereby improving the processing efficiency and accuracy of data processing by multi-modal large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 An application scenario diagram of the feature data compression method provided in the embodiment of the present disclosure;

[0020] Figure 2 Schematic diagram of the process of the characteristic data compression method provided in the embodiment of the present disclosure Figure 1 ;

[0021] Figure 3 for Figure 2A schematic diagram of specific steps for generating enhanced query data corresponding to step S102 in the illustrated embodiment;

[0022] Figure 4 A schematic diagram of a process of processing an initial feature sequence to obtain global feature data provided by an embodiment of the present disclosure;

[0023] Figure 5 for Figure 3 A schematic diagram of specific steps for generating enhanced query data corresponding to step S1022 in the illustrated embodiment;

[0024] Figure 6 for Figure 2 A schematic diagram of specific steps for generating formatted feature data corresponding to step S102 in the illustrated embodiment;

[0025] Figure 7 A schematic diagram of a process of processing an initial feature sequence to obtain formatted feature data provided by an embodiment of the present disclosure;

[0026] Figure 8 A schematic diagram of a process for generating and applying compressed feature data provided by an embodiment of the present disclosure;

[0027] Fig. 9 Schematic diagram of the characteristic data compression method provided in the embodiment of the present disclosure Figure 2 ;

[0028] Fig.10 for Fig. 9 A schematic diagram of specific implementation steps of step S203 in the illustrated embodiment;

[0029] Fig.11 A schematic diagram of a process for processing data based on multiple cross-attention layers to obtain compressed feature data provided in an embodiment of the present disclosure;

[0030] Fig.12 A structural block diagram of a feature data compression device provided in an embodiment of the present disclosure;

[0031] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;

[0032] Fig.14 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] The application scenarios of the embodiments of the present disclosure are explained below:

[0036] Figure 1 An application scenario diagram of the feature data compression method provided in the embodiment of the present disclosure. The feature data compression method provided in the embodiment of the present disclosure can be applied to the application scenario of parsing videos. More specifically, it can be applied to the scenario of parsing videos based on a multimodal large model, wherein the multimodal large model is deployed in an application (APP, Application) with a video parsing function, such as a video processing application, an artificial intelligence application, etc. The execution subject of this embodiment can be a terminal device running the above-mentioned application with a video parsing function deployed with a multimodal large model, or it can be a server on the server side corresponding to the above-mentioned application, or other electronic devices that perform similar functions.

[0037] Among them, in some embodiments, the terminal device or server can implement the characteristic data compression method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be program-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local application, that is, a program that needs to be installed in the operating system to run, or it can be a small program embedded in any APP, that is, a program running based on a browser environment. In summary, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form, and the specific implementation form can be configured as needed. Further, in some embodiments, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud storage, cloud communication, cloud database, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms and other basic cloud computing services, wherein the cloud service can be an interactive processing service for terminal devices to call.

[0038] refer to Figure 1 As shown in , taking the case where the server is the execution subject as an example, an application with video parsing function (hereinafter referred to as the application) is running in the terminal device, that is, the client of the application is installed on the terminal device, and the server of the application is deployed on the server. The multimodal large model provided by the server is used to implement the application's video parsing processing to generate the video information required by the user. Specifically, refer to Figure 1As shown in, after the video to be processed and the user instruction text are loaded through the application, by triggering the corresponding functional components, such as the "analysis" component shown in the figure, the terminal device sends the video to be processed and the user instruction text to the server side for processing. On the one hand, the server encodes the video to be processed by calling the visual encoder to generate a video feature sequence. On the other hand, the server encodes the user instruction text by calling the text encoder to generate user instruction features. Then, the server generates video description information based on the video feature sequence and the user instruction feature by calling the large language model, and sends the video description information back to the terminal device, and displays it on the terminal device side, so that the application responds to the user instruction text input by the user, analyzes the video to be processed, and generates the video description information required by the user. Of course, it can be understood that in another possible implementation, the execution subject of the method provided in this embodiment can also be the terminal device itself, that is, the above-mentioned processing process is all executed by the terminal device, and the models and algorithms used in the execution process can be all deployed locally on the terminal device, or partially deployed locally on the terminal device, and partially deployed on an external device and called by the terminal device. The specific implementation process is similar and will not be repeated this time.

[0039] In the related art, when a video is processed by a large multimodal model, the feature data of the video to be processed is usually input into the large language model for processing, for example Figure 1 In the application scenario shown, after the server receives the video to be processed and the user command text sent by the terminal device, the server calls the visual encoder to encode the video to be processed, obtains a video feature sequence, that is, the feature data of the video to be processed, and calls the text encoder to encode the user command text to generate user command features. Then, the large language model can generate video description information based on the feature data (video feature sequence) and the user command features, that is, the video information required by the user. However, the feature data processed by the large language model is the encoded data directly generated by the visual encoder, that is, the feature data is the data generated by encoding the video to be processed. Since the video to be processed includes time features, spatial features and channel features, the time features are used to indicate the playback duration of the video to be processed, the spatial features are used to indicate the image features of the video frames of the video to be processed, and the channel features are used to indicate the color features of the image features. Based on the above three features of the video to be processed, the amount of feature data generated by the visual encoder for the video to be processed is too large. That is, in the solution of the prior art, there is a problem of too much feature data processed by the large language model. At the same time, too much feature data will increase the computational complexity of the large language model, thereby causing the large language model to operate inefficiently, generate video information required by users slowly, and have a poor user experience.

[0040] The embodiment of the present disclosure provides a feature data compression method to solve the above problem.

[0041] refer to Figure 2 , Figure 2 Schematic diagram of the process of the characteristic data compression method provided in the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a server, a terminal device or other electronic devices. The characteristic data compression method includes:

[0042] Step S101: obtaining an initial feature sequence of a target video, where the initial feature sequence represents video content features of the target video.

[0043] refer to Figure 1 As shown in the schematic diagram of the application scenario, in this embodiment, the server is used as the execution subject to introduce the provided feature data compression method. For example, the server is deployed with a server corresponding to an application with a video parsing function, and the terminal device runs the client of the application. The server and the terminal device communicate based on the server-client, and then obtain the target video sent by the terminal device. The target video is Figure 1 The video to be processed in the application scenario shown is the video uploaded by the client installed in the terminal device. After receiving the target video sent by the terminal device, the server calls the visual encoder to encode the target video and obtains the initial feature sequence of the target video. That is, the video content features of the target video are represented by the initial feature sequence. The initial feature sequence is Figure 1 Video feature sequences of the video to be processed introduced in the shown scenario.

[0044] Furthermore, the video content features of the target video include time features, space features and channel features. According to the video content features of the target video, the feature dimension components under the corresponding features can be obtained, that is, the initial feature sequence includes time dimension components, space dimension components and channel dimension components. The time dimension components correspond to the time features, the space dimension components correspond to the space features, and the channel dimension components correspond to the channel features. Among them, the time dimension components are used to characterize the playback time sequence of the target video, the space dimension components are used to characterize the image features of the video frames corresponding to each timestamp in the playback time sequence, and the channel dimension components are used to characterize the number of feature channels of the image features. Specifically, for example, if the target video has a duration of 30 seconds and a frame rate of 30 frames per second, then the corresponding time dimension component is 900 frames, each of which corresponds to a specific time point of the target video, and these 900 frames together constitute the playback time sequence of the target video; the spatial dimension component of the target video corresponds to the image features of each of the 900 video frames, and the image features include information about objects in each frame of the video image, such as the shape information, size information, and position information of the object. For example, if the video resolution of the target video is 1080P (e.g., 1920*1080 pixels), then the spatial dimension component of the target video is 1920*1080, and for each video frame of the target video, it is Composed of 1920 horizontal pixels and 1080 vertical pixels, the total number of pixels of this video frame is 2073600 pixels (the product of 1920 and 1080), and these 2073600 pixels together constitute the image features of a single video frame; the channel dimension component of the target video is used to indicate the color features of the video image. For example, if the target video is a color video, then for a video frame (RGB image) in the target video, its channel dimension component is 3, namely the three channels of red, green and blue. Furthermore, for a single video frame, the color of each pixel is determined by the brightness values ​​of the three channels of red, green and blue, that is, the channel dimension component determines the color features of the image features of the video frame.

[0045] Step S102: According to the initial feature sequence, enhanced query data and formatted feature data are generated respectively, wherein the enhanced query data is constructed based on the global feature data, the global feature data is the data after the time dimension of the initial feature sequence is reduced, and the formatted feature data is the data after the initial feature sequence is reformatted based on the target data format.

[0046] Exemplarily, the initial feature sequence is reduced in dimension in the time dimension, that is, the server compresses the target video in the time dimension to obtain a two-dimensional image including spatial dimension and channel dimension, that is, global feature data; then, based on the spatial dimension information and channel dimension information corresponding to the global feature data, a feature vector of the two-dimensional image is constructed, and then the feature vector is projected into a low-dimensional space to obtain an initial query vector for the two-dimensional image, and further based on the obtained initial query vector, enhanced query data can be constructed.

[0047] Exemplarily, based on a preset target data format, the initial feature sequence is reformatted to obtain formatted feature data. Specifically, for example, the target data format is a data format that does not lose the video content features of the target video, and then the server integrates the data format of the initial feature sequence based on the target data format while retaining the video content features, thereby obtaining reformatted data, that is, obtaining formatted feature data.

[0048] Further, refer to Figure 3 , Figure 3 for Figure 2 The schematic diagram of specific steps for generating enhanced query data corresponding to step S102 in the illustrated embodiment, since the initial feature sequence includes a time dimension component, a space dimension component and a channel dimension component, the specific implementation steps for generating enhanced query data according to the initial feature sequence include:

[0049] Step S1021: averaging the initial feature sequence in the time dimension component to obtain global feature data including only the space dimension component and the channel dimension component;

[0050] Step S1022: Obtain enhanced query data based on the global feature data.

[0051] Exemplarily, the server determines the specific data features of the time dimension component included in the initial feature sequence based on the video duration and video frame rate of the target video, and then averages the initial feature sequence based on the specific data features corresponding to the time dimension component to obtain global feature data that only includes the space dimension component and the channel dimension component. Furthermore, based on the space dimension component and the channel dimension component corresponding to the global feature data, a corresponding target feature vector is constructed, and by performing dimensionality reduction processing on the target feature vector, a corresponding initial query vector is obtained, that is, enhanced query data is obtained. Specifically, for example, referring to Figure 4 , Figure 4A schematic diagram of a process for processing an initial feature sequence to obtain global feature data is provided for an embodiment of the present disclosure. For example, the video length of a target video is 10 seconds, and the video frame rate of the target video is 30 frames / second. The time dimension component corresponding to the initial feature sequence of the target video is 300 frames, that is, the target video has 300 video frames, that is, the target video corresponds to 300 two-dimensional images. Then, the server processes the initial feature sequence based on the specific data features corresponding to the 300 video frames, for example, superimposing and merging the 300 video frames to obtain a two-dimensional image, and the image feature data in the two-dimensional image is the global feature data. By way of example, for example, the target video is a video of a small ball moving, and the two-dimensional image finally obtained based on the superposition and merging of the time dimension components is an image of the movement path of the small ball.

[0052] In the steps of this embodiment, by averaging the initial feature sequence in the time dimension component, the redundant information in the initial feature sequence is eliminated, the amount of data for processing global feature data in subsequent steps is reduced, and fusion processing with other data is facilitated, thereby improving the model processing performance and processing efficiency.

[0053] Furthermore, in another possible implementation, referring to Figure 5 , Figure 5 for Figure 3 In the illustrated embodiment, the specific steps of generating enhanced query data corresponding to step S1022 include:

[0054] Step S10221: Create an initial query feature according to the feature sizes of the spatial dimension components and the channel dimension components. The initial query feature is a learnable feature matrix of all zeros.

[0055] Step S10222: Obtain enhanced query data based on the sum of the initial query features and the global feature data.

[0056] Exemplarily, the spatial dimension component is used to indicate the image features of the video frame corresponding to each timestamp of the target video, that is, as described above, based on the spatial dimension component, the image information of a single video frame can be determined, and based on the video resolution of the target video, the number of pixels of a single video frame corresponding to the spatial dimension component can be determined, that is, the total number of pixels of a video frame with a video resolution of 1080P (1920*1080 pixels) is the product of 1920 and 1080, that is, 2073600 pixels, that is, the characteristic size of the spatial dimension component is 2073600, which is jointly determined by the horizontal pixels and vertical pixels of the target video; further, the channel dimension The component is used to characterize the number of feature channels of the image features, that is, the feature size of the channel dimension component corresponds to the number of feature channels of the image features. For example, if the target video is an RGB color video, the number of feature channels of the image features of the RGB color video is 3, that is, the channel dimension component is 3, that is, the feature size is 3, wherein these three channels are the red channel, the green channel and the blue channel; or, if the target video is a grayscale video, the number of feature channels of the image features of the grayscale video is 1, that is, the channel dimension component is 1, that is, the feature size is 3, wherein this single channel is the brightness channel, and this brightness channel contains the light and dark information in the grayscale video.

[0057] Furthermore, based on the feature size of the spatial dimension component and the feature size of the channel dimension component, an initial query feature can be created, wherein the initial query feature is a learnable feature matrix of all zeros. Specifically, for example, according to the characteristic size of the spatial dimension component of the target video with a video resolution of 1080P (1920*1080 pixels), it corresponds to 1920 horizontal pixels and 1080 vertical pixels, and then based on the characteristic size of the spatial dimension component determined above (horizontal pixels and vertical pixels), the size of the all-zero learnable feature matrix in the spatial dimension is obtained, that is, the number of rows and columns of the all-zero learnable feature matrix is ​​obtained, wherein the number of rows of the all-zero learnable feature matrix is ​​related to the number of vertical pixels, and the number of columns of the all-zero learnable feature matrix is ​​related to the number of horizontal pixels; further, if the target video is an RGB color video, then its characteristic size in the channel dimension component is 3, and the depth of the all-zero learnable feature matrix is ​​3; if the target video is a grayscale video, then its characteristic size in the channel dimension component is 1, and the depth of the all-zero learnable feature matrix is ​​1.

[0058] Furthermore, by summing up the all-zero learnable feature matrix corresponding to the initial query feature and the target feature vector corresponding to the global feature data, enhanced query data can be obtained. Specifically, since both the initial query feature and the global feature data are obtained from the initial feature sequence of the target video, and more specifically, both are determined based on the spatial dimension component and the channel dimension component, the number of rows and columns of the matrices corresponding to the two are consistent, and thus, the corresponding elements of the two can be added together to obtain enhanced query data.

[0059] In the steps of this embodiment, the initial query features are enhanced based on the global feature data of the target video, which avoids the initial query features being general data when processing the formatted feature data in subsequent steps, thereby achieving effective feature compression of the target video.

[0060] Further, refer to Figure 6 , Figure 6 for Figure 2 The schematic diagram of specific steps for generating formatted feature data corresponding to step S102 in the illustrated embodiment, since the initial feature sequence includes a time dimension component, a space dimension component and a channel dimension component, the specific implementation steps for generating formatted feature data according to the initial feature sequence include:

[0061] Step S102A: Fusing the time dimension component and the space dimension component of the initial feature sequence to obtain the space-time dimension component.

[0062] Step S102B: Generate formatted feature data according to the spatiotemporal dimension components and the channel dimension components.

[0063] Exemplarily, the preset target data format is a data format based on the time-space dimension and the channel dimension. First, the time dimension and the space dimension are integrated for the initial feature sequence, that is, the time dimension component and the space dimension component of the initial feature sequence are processed to obtain the time-space dimension component, and then the time-space dimension component and the channel dimension component of the initial feature sequence are processed to obtain the feature data format based on the time-space dimension and the channel dimension. Specifically, refer to Figure 7 , Figure 7A schematic diagram of a process for processing an initial feature sequence to obtain formatted feature data is provided for an embodiment of the present disclosure. For example, the video length of a target video is 10 seconds, the video frame rate of the target video is 30 frames per second, and the video resolution of the target video is 1080P (1920*1080 pixels). The time dimension component corresponding to the initial feature sequence of the target video is 300 frames, that is, the target video has 300 video frames, that is, the target video corresponds to 300 two-dimensional images; then, based on the space dimension component corresponding to the initial feature sequence of the target video, the image features of each of the 300 video frames are determined; further, the 300 video frames are fused according to the time dimension component and the space dimension component corresponding to the target video, so as to obtain the space-time dimension component of the target video, wherein one of the forms of the space-time dimension component is a matrix, and the matrix is ​​related to Figure 7 The video frame splicing diagram shown in the figure is obtained by fusion and splicing of 300 video frames of the target video; further, the spatiotemporal dimension components and the channel dimension components of the initial feature sequence are processed, that is, the matrix corresponding to the spatiotemporal dimension components is assigned the characteristics of the channel dimension, and formatted feature data can be obtained, that is, the color characteristics of the video image corresponding to the channel dimension components are fused to the image characteristics corresponding to the spatiotemporal dimension components to ensure that the video content characteristics of the target video are not lost, that is, the formatted feature data retains all the video content characteristics of the target video.

[0064] Step S103: Through the feature enhancement model, the enhanced query data and the formatted feature data are processed separately to obtain compressed feature data of the target video, wherein the feature enhancement model includes multiple cross-attention layers, and the multiple cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate compressed feature data.

[0065] Exemplarily, since both the enhanced query data and the formatted feature data are obtained by processing the initial feature sequence, there is a correlation between the enhanced query data and the formatted feature data, and then the server gradually extracts the key features in the formatted feature data through multiple cross-attention layers according to the correlation between the enhanced query data and the formatted feature data, so as to obtain the compressed feature data of the target video, that is, the server processes the enhanced query data and the formatted feature data through the feature enhancement model to generate compressed feature data, wherein the feature enhancement model includes multiple cross-attention layers. Specifically, the cross-attention layer processes the enhanced query data and the formatted feature data including: determining the corresponding query vector based on the enhanced query data, determining the corresponding key vector based on the formatted feature data, and then obtaining the key features in the formatted feature data based on the query vector and the key vector, and then processing the above data and / or key features based on multiple cross-attention layers to obtain the compressed feature data of the target video.

[0066] In one possible implementation, reference Figure 8 , Figure 8 A schematic diagram of a process for generating and applying compressed feature data provided by an embodiment of the present disclosure. Specifically, Figure 8 As shown, taking the case where the server is the execution subject as an example, an application Deal_Video (hereinafter referred to as the application Deal_Video) with a video parsing function is running in the terminal device, and the user uploads the target video and the user instruction text through the application Deal_Video installed on the terminal device, and then the terminal device sends the target video and the user instruction text to the server; further, after receiving the target video and the user instruction text sent by the terminal device, the server calls the multimodal large model to process the target video and the user instruction text respectively, wherein in the multimodal large model, the text encoder is called to encode the user instruction text , generate user command features, call the visual encoder and the feature enhancement model to encode the target video, generate compressed feature data of the target video, and then process the compressed feature data and the user command features based on the large language model to obtain the target video information required by the user, and send the target video information back to the terminal device and display it on the terminal device side, wherein the target video information is used to indicate the content obtained by parsing the target video, for example, the target video information is specifically a text description of the target video corresponding to the user command text (for example: summary, full, partial), and / or at least one frame of image obtained by key frame extraction of the above target video.

[0067] Furthermore, refer to Figure 8The specific process of "calling the visual encoder and the feature enhancement model to encode the target video and generate compressed feature data of the target video" includes: calling the visual encoder to encode the target video and generate a corresponding initial feature sequence, wherein the initial feature sequence includes a time dimension component, a space dimension component and a channel dimension component, and then the server reduces the dimension of the initial feature sequence in the time dimension by calling the data processing model, that is, compresses the time dimension component to obtain global feature data, and constructs the global feature data to obtain enhanced query data; at the same time or asynchronously, the server reshapes the initial feature sequence according to a preset target data format by calling the data processing model to obtain formatted feature data; further, the server calls the feature enhancement model to realize processing and analysis of the correlation between the enhanced query data and the formatted feature data through multiple cross-attention layers, extracts key features in the formatted feature data, and generates compressed feature data of the target video based on the key features.

[0068] In this embodiment, an initial feature sequence of a target video is obtained, the initial feature sequence characterizing the video content features of the target video; based on the initial feature sequence, enhanced query data and formatted feature data are generated respectively, wherein the enhanced query data is constructed based on global feature data, the global feature data is data after dimension reduction is performed on the time dimension of the initial feature sequence, and the formatted feature data is data after formatting the initial feature sequence based on the target data format; the enhanced query data and the formatted feature data are processed respectively through a feature enhancement model to obtain compressed feature data of the target video, wherein the feature enhancement model includes a plurality of cross-attention layers, and the plurality of cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate compressed feature data. By utilizing the initial feature sequence, enhanced query data constructed by global feature data is generated, and then the enhanced query data and the formatted feature data generated after formatting the initial feature sequence are processed through a cross-attention layer to obtain compressed feature data. Since the enhanced query data contains global feature data that is reduced in dimension based on the time dimension, video features are gradually extracted and compressed based on it through multiple layers of cross-attention layers to achieve continuous attention to key features. On the one hand, the data volume of key features extracted through cross-attention is reduced. On the other hand, the information in the global feature data is utilized to improve the accuracy of the obtained key features, and the feature validity of the compressed feature data finally output is improved, which solves the problem of excessive data volume of feature data of the video to be processed (target video), thereby improving the processing efficiency and accuracy of data processing by multi-modal large models.

[0069] refer to Fig. 9 , Fig. 9Schematic diagram of the characteristic data compression method provided in the embodiment of the present disclosure Figure 2 In this embodiment Figure 2 On the basis of the embodiment shown, step S103 is further refined, and the characteristic data compression method includes:

[0070] Step S201: obtaining an initial feature sequence of a target video, where the initial feature sequence represents video content features of the target video.

[0071] Step S202: According to the initial feature sequence, enhanced query data and formatted feature data are generated respectively, wherein the enhanced query data is constructed based on the global feature data, the global feature data is the data after the time dimension of the initial feature sequence is reduced, and the formatted feature data is the data after the initial feature sequence is reformatted based on the target data format.

[0072] Step S203: Process the corresponding enhanced query data and formatted feature data through the current cross-attention layer to obtain the key features corresponding to the current cross-attention layer.

[0073] Exemplarily, the cross-attention layer processes the enhanced query data and the formatted feature data by: determining the corresponding query vector based on the enhanced query data, determining the corresponding key vector and value vector based on the formatted feature data, and then calculating the similarity between the query vector and the key vector, and calculating in combination with the value vector, to obtain the key features of the target video output by the current cross-attention layer. Specifically, for example, the formatted feature data includes a key sequence and a value sequence, refer to Fig.10 , Fig.10 for Fig. 9 The specific implementation steps of step S203 in the embodiment shown in the figure include:

[0074] Step S2031: linearly transform the enhanced query data, key sequence and value sequence respectively to obtain corresponding query vectors, key vectors and value vectors.

[0075] Step S2032: Obtain a scaling factor according to the vector dimension of the key vector.

[0076] Step S2033: Obtain an attention score matrix based on the product of the query vector and the key vector and the ratio of the scaling factor.

[0077] Step S2034: Use the normalized exponential function to process the attention score matrix to obtain the attention weight matrix.

[0078] Step S2035: Obtain key features based on the product of the attention weight matrix and the value vector.

[0079] For example, the enhanced query data is Q1, and the formatted feature data includes a key sequence K1 and a value sequence V1. Then the corresponding query vector Key Vector Sum value vector From formula (1), we can get

[0080]

[0081] Among them, Linear1, Linear2 and Linear3 correspond to different or the same linear transformation algorithms respectively;

[0082] Furthermore, the scaling factor X is obtained based on the vector dimension of the key vector, as shown in formula (2),

[0083]

[0084] Where dk is the key vector The vector dimension of

[0085] Furthermore, the attention score matrix It is obtained by multiplying the query vector and the key vector by the ratio of the scaling factor, as shown in formula (3).

[0086]

[0087] Furthermore, the attention weight matrix is ​​obtained by transforming the attention score matrix The normalized exponential function is processed to obtain, as shown in formula (4),

[0088]

[0089] Among them, Softmax is a normalized exponential function;

[0090] Furthermore, the key feature Attention is obtained by multiplying the attention weight matrix and the value vector, as shown in formula (5):

[0091]

[0092] Step S204: Determine whether the current cross-attention layer is the last cross-attention layer in the feature enhancement model.

[0093] Step S205: If the current cross-attention layer is the last cross-attention layer in the feature enhancement model, the key features are output as compressed feature data.

[0094] Step S206: If the current cross-attention layer is not the last cross-attention layer in the feature enhancement model, based on the key features and the query feature data corresponding to the next cross-attention layer, the enhanced query data corresponding to the next cross-attention layer is generated by fusion, and the next cross-attention layer is set as the current cross-attention layer, and the process returns to execute step S203.

[0095] For example, reference Fig.11 , Fig.11 A schematic diagram of a process for processing data based on multiple cross-attention layers to obtain compressed feature data provided in an embodiment of the present disclosure, such as Fig.11 As shown, for example, there are k cross attention layers, which generate global feature data and formatted feature data according to the initial feature sequence of the target video, and then construct enhanced query data data_1 according to the global feature data, such as Fig.11 As shown in, according to the global feature data and the initial query feature (query feature data) find_1, the enhanced query data data_1 is constructed, and the first cross-attention layer layer_1 generates the key feature key_1 according to the formatted feature data and the enhanced query data data_1. The initial query feature find_1 corresponds to the first cross-attention layer layer_1; then, the query feature data corresponding to the second cross-attention layer layer_2 is find_2. According to the key feature key_1 and the query feature data find_2, the enhanced query data data_2 is constructed, and the second cross-attention layer layer_2 generates the key feature key_1 according to the formatted feature data and the enhanced query data data_1. The feature data and the enhanced query data data_2 are formatted to generate the key feature key_2; until the (k-1)th cross-attention layer layer_(k-1) generates the key feature key_(k-1), and then the query feature data corresponding to the kth cross-attention layer layer_k is find_k, and according to the key feature key_(k-1) and the query feature data find_k, the enhanced query data data_k is constructed, and the kth cross-attention layer layer_k generates the key feature key_k according to the formatted feature data and the enhanced query data data_k, that is, the key feature key_k is output as the compressed feature data of the target video.

[0096] Furthermore, based on the compressed feature data output by the feature enhancement model, the large language model combines the user command features to generate the target video information required by the user; and further, based on user feedback and / or the residual corresponding to the compressed feature data, the query feature data corresponding to all the above-mentioned cross-attention layers are updated and optimized through the back propagation algorithm and gradient descent (or other optimization algorithms).

[0097] In the steps of this embodiment, the query feature data of the next cross-attention layer is fused according to the key features output by the previous cross-attention layer to generate enhanced query data strongly correlated with the target video, and then the formatted feature data is further processed by the cross-attention layer based on the enhanced query data to generate the key features of this layer. This cycle is repeated until the last cross-attention layer, thereby enhancing the query feature data corresponding to each cross-attention layer based on the global feature data of the target video, avoiding the query feature data being general data when processing the formatted feature data, and fully parsing the effective information of different levels in the formatted feature data by stacking multiple cross-attention layers, thereby achieving effective feature compression of the target video.

[0098] Further, in Fig.11 In the illustrated embodiment, the number of the multiple cross-attention layers is a preset value; in another possible implementation, the number of the multiple cross-attention layers is determined based on the target hyperparameters of the feature enhancement model; specifically, the target hyperparameters of the feature enhancement model are obtained; and the number of cross-attention layers in the feature enhancement model is determined according to the target hyperparameters. Exemplarily, the target hyperparameters of the feature enhancement model are determined according to the initial feature sequence of the target video uploaded by the user, that is, the complexity of the target video is determined based on the video content features of the target video, and then the corresponding target hyperparameters are determined according to the different complexities of the target video and the preset matching relationship, and then the number of cross-attention layers in the feature enhancement model is determined according to the determined target hyperparameters. Alternatively, the target hyperparameters of the feature enhancement model are determined based on the initial feature sequence of the target video uploaded by the user and the user command features corresponding to the user command text, that is, the corresponding target hyperparameters are determined based on the different complexities of the target video and the different complexities of the user requirements for parsing the target video corresponding to the user command text, and then the number of cross-attention layers in the feature enhancement model is determined based on the determined target hyperparameters; the number of cross-attention layers in the feature enhancement model determined based on the target hyperparameters that match the target video not only improves the processing efficiency of the feature enhancement model in processing data, but also improves the generalization ability of the feature enhancement model and reduces the risk of overfitting.

[0099] In this embodiment, the implementation method of step 201-step 202 is the same as that of the present disclosure. Figure 2 The implementation methods of step 101 and step 102 in the illustrated embodiment are the same and will not be described in detail here.

[0100] Corresponding to the feature data compression method of the above embodiment, Fig.12This is a structural block diagram of a feature data compression device provided in an embodiment of the present disclosure. The method described in the above embodiment can be executed by the feature data compression device, which can be implemented by software and / or hardware, and can be integrated in an electronic device with certain data processing functions. The electronic device may include but is not limited to a mobile terminal with big data processing capabilities, as well as a fixed terminal with big data processing capabilities such as a desktop computer and a supercomputer.

[0101] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Fig.12 , the characteristic data compression device 3 comprises:

[0102] An acquisition module 31 is used to acquire an initial feature sequence of a target video, where the initial feature sequence represents video content features of the target video;

[0103] A generating module 32, for generating enhanced query data and formatted feature data according to the initial feature sequence, wherein the enhanced query data is constructed based on the global feature data, the global feature data is data after dimension reduction for the time dimension of the initial feature sequence, and the formatted feature data is data after formatting the initial feature sequence based on the target data format;

[0104] The processing module 33 is used to process the enhanced query data and the formatted feature data respectively through the feature enhancement model to obtain the compressed feature data of the target video, wherein the feature enhancement model includes multiple cross-attention layers, and the multiple cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate compressed feature data.

[0105] According to one or more embodiments of the present disclosure, the initial feature sequence includes a time dimension component, a space dimension component and a channel dimension component, wherein the time dimension component is used to characterize the playback time sequence of the target video, the space dimension component is used to characterize the image features of the video frames corresponding to each timestamp in the playback time sequence, and the channel dimension component is used to characterize the number of feature channels of the image features.

[0106] According to one or more embodiments of the present disclosure, when the generation module 32 generates enhanced query data based on the initial feature sequence, it is specifically used to: average the initial feature sequence on the time dimension component to obtain global feature data that only includes space dimension components and channel dimension components; and obtain enhanced query data based on the global feature data.

[0107] According to one or more embodiments of the present disclosure, the generation module 32 is further used to: create an initial query feature based on the feature sizes of the spatial dimension components and the channel dimension components, and the initial query feature is a learnable feature matrix of all zeros; when the generation module 32 obtains the enhanced query data based on the global feature data, it is specifically used to: obtain the enhanced query data based on the sum of the initial query feature and the global feature data.

[0108] According to one or more embodiments of the present disclosure, when the generation module 32 generates formatted feature data based on the initial feature sequence, it is specifically used to: fuse the time dimension component and the space dimension component of the initial feature sequence to obtain the space-time dimension component; and generate formatted feature data based on the space-time dimension component and the channel dimension component.

[0109] According to one or more embodiments of the present disclosure, when the processing module 33 processes the enhanced query data and the formatted feature data respectively through the feature enhancement model to obtain the compressed feature data of the target video, it is specifically used to: loop the following steps: process the corresponding enhanced query data and the formatted feature data through the current cross-attention layer to obtain the key features corresponding to the current cross-attention layer; if the current cross-attention layer is the last cross-attention layer in the feature enhancement model, the key features are output as compressed feature data; if the current cross-attention layer is not the last cross-attention layer in the feature enhancement model, based on the key features and the query feature data corresponding to the next cross-attention layer, the enhanced query data corresponding to the next cross-attention layer is generated by fusion, and the next cross-attention layer is set as the current cross-attention layer.

[0110] According to one or more embodiments of the present disclosure, the processing module 33 is further used to: obtain target hyperparameters of the feature enhancement model; and determine the number of cross-attention layers in the feature enhancement model based on the target hyperparameters.

[0111] According to one or more embodiments of the present disclosure, the formatted feature data includes a key sequence and a value sequence. When the processing module 33 processes the corresponding enhanced query data and the formatted feature data through the current cross-attention layer to obtain the key features corresponding to the current cross-attention layer, it is specifically used to: perform linear transformation on the enhanced query data, the key sequence and the value sequence respectively to obtain the corresponding query vector, the key vector and the value vector; obtain the scaling factor according to the vector dimension of the key vector; obtain the attention score matrix according to the product of the query vector and the key vector and the ratio of the scaling factor; process the attention score matrix using a normalized exponential function to obtain the attention weight matrix; obtain the key features according to the product of the attention weight matrix and the value vector.

[0112] The acquisition module 31, the generation module 32 and the processing module 33 are connected in sequence. The characteristic data compression device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0113] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Fig.13 As shown, the electronic device 4 includes:

[0114] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0115] The memory 42 stores computer executable instructions;

[0116] The processor 41 executes the computer execution instructions stored in the memory 42 to implement the following Figure 2-Figure 11 The characteristic data compression method in the illustrated embodiment.

[0117] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0118] For related instructions, please refer to Figure 2-Figure 11 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.

[0119] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement the present invention when executed by a processor. Figure 2-Figure 11 The feature data compression method provided by any one of the corresponding embodiments.

[0120] The present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the present invention is realized. Figure 2-Figure 11 The feature data compression method provided by any one of the corresponding embodiments.

[0121] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0122] refer to Fig.14 , Fig.14The hardware structure diagram of the electronic device provided in the embodiment of the present disclosure shows a schematic diagram of the structure of the electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 can be a terminal device or a server. The terminal device can include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.14 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0123] like Fig.14 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0124] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.14 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0125] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0126] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0127] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0128] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0129] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit or module does not, in some cases, constitute a limitation on the unit itself.

[0132] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] In a first aspect, according to one or more embodiments of the present disclosure, a feature data compression method is provided, comprising:

[0135] An initial feature sequence of a target video is obtained, wherein the initial feature sequence characterizes the video content features of the target video; based on the initial feature sequence, enhanced query data and formatted feature data are generated respectively, wherein the enhanced query data is constructed based on global feature data, the global feature data is data after dimension reduction is performed on the time dimension of the initial feature sequence, and the formatted feature data is data after the initial feature sequence is reformatted based on the target data format; the enhanced query data and the formatted feature data are processed respectively through a feature enhancement model to obtain compressed feature data of the target video, wherein the feature enhancement model includes a plurality of cross-attention layers, and the plurality of cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate the compressed feature data.

[0136] According to one or more embodiments of the present disclosure, the initial feature sequence includes a time dimension component, a space dimension component and a channel dimension component, wherein the time dimension component is used to characterize the playback time sequence of the target video, the space dimension component is used to characterize the image features of the video frames corresponding to each timestamp in the playback time sequence, and the channel dimension component is used to characterize the number of feature channels of the image features.

[0137] According to one or more embodiments of the present disclosure, generating enhanced query data based on the initial feature sequence includes: averaging the initial feature sequence on the time dimension component to obtain global feature data that only includes the space dimension component and the channel dimension component; and obtaining the enhanced query data based on the global feature data.

[0138] According to one or more embodiments of the present disclosure, the method further includes: creating an initial query feature based on the feature sizes of the spatial dimension component and the channel dimension component, the initial query feature being a learnable feature matrix of all zeros; and obtaining enhanced query data based on the global feature data, including: obtaining enhanced query data based on the sum of the initial query feature and the global feature data.

[0139] According to one or more embodiments of the present disclosure, generating formatted feature data based on the initial feature sequence includes: fusing the time dimension component and the space dimension component of the initial feature sequence to obtain the space-time dimension component; and generating the formatted feature data based on the space-time dimension component and the channel dimension component.

[0140] According to one or more embodiments of the present disclosure, the enhanced query data and the formatted feature data are processed separately through the feature enhancement model to obtain the compressed feature data of the target video, including: looping the following steps: processing the corresponding enhanced query data and the formatted feature data through the current cross-attention layer to obtain the key features corresponding to the current cross-attention layer; if the current cross-attention layer is the last cross-attention layer in the feature enhancement model, outputting the key features as the compressed feature data; if the current cross-attention layer is not the last cross-attention layer in the feature enhancement model, based on the key features and the query feature data corresponding to the next cross-attention layer, the enhanced query data corresponding to the next cross-attention layer is generated by fusion, and the next cross-attention layer is set as the current cross-attention layer.

[0141] According to one or more embodiments of the present disclosure, the method further includes: obtaining target hyperparameters of the feature enhancement model; and determining the number of cross-attention layers in the feature enhancement model based on the target hyperparameters.

[0142] According to one or more embodiments of the present disclosure, the formatted feature data includes a key sequence and a value sequence, and the corresponding enhanced query data and the formatted feature data are processed through the current cross-attention layer to obtain the key features corresponding to the current cross-attention layer, including: linearly transforming the enhanced query data, the key sequence and the value sequence respectively to obtain corresponding query vectors, key vectors and value vectors; obtaining a scaling factor according to the vector dimension of the key vector; obtaining an attention score matrix according to the product of the query vector and the key vector and the ratio of the scaling factor; processing the attention score matrix using a normalized exponential function to obtain an attention weight matrix; obtaining the key features according to the product of the attention weight matrix and the value vector.

[0143] In a second aspect, according to one or more embodiments of the present disclosure, a feature data compression device is provided, comprising:

[0144] An acquisition module, used to acquire an initial feature sequence of a target video, wherein the initial feature sequence represents video content features of the target video;

[0145] A generating module, configured to generate enhanced query data and formatted feature data according to the initial feature sequence, wherein the enhanced query data is constructed based on global feature data, the global feature data is data after dimension reduction is performed on the time dimension of the initial feature sequence, and the formatted feature data is data after the initial feature sequence is formatted based on a target data format;

[0146] A processing module is used to process the enhanced query data and the formatted feature data respectively through a feature enhancement model to obtain compressed feature data of the target video, wherein the feature enhancement model includes a plurality of cross-attention layers, and the plurality of cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate the compressed feature data.

[0147] According to one or more embodiments of the present disclosure, the initial feature sequence includes a time dimension component, a space dimension component and a channel dimension component, wherein the time dimension component is used to characterize the playback time sequence of the target video, the space dimension component is used to characterize the image features of the video frames corresponding to each timestamp in the playback time sequence, and the channel dimension component is used to characterize the number of feature channels of the image features.

[0148] According to one or more embodiments of the present disclosure, when the generation module generates enhanced query data based on the initial feature sequence, it is specifically used to: average the initial feature sequence on the time dimension component to obtain global feature data that only includes the space dimension component and the channel dimension component; and obtain the enhanced query data based on the global feature data.

[0149] According to one or more embodiments of the present disclosure, the generation module is further used to: create an initial query feature according to the feature sizes of the spatial dimension component and the channel dimension component, and the initial query feature is a learnable feature matrix of all zeros; when the generation module obtains the enhanced query data according to the global feature data, it is specifically used to: obtain the enhanced query data according to the sum of the initial query feature and the global feature data.

[0150] According to one or more embodiments of the present disclosure, when the generation module generates formatted feature data based on the initial feature sequence, it is specifically used to: fuse the time dimension component and the space dimension component of the initial feature sequence to obtain the space-time dimension component; and generate the formatted feature data based on the space-time dimension component and the channel dimension component.

[0151] According to one or more embodiments of the present disclosure, when the processing module processes the enhanced query data and the formatted feature data respectively through the feature enhancement model to obtain the compressed feature data of the target video, it is specifically used to: loop the following steps: process the corresponding enhanced query data and the formatted feature data through the current cross-attention layer to obtain the key features corresponding to the current cross-attention layer; if the current cross-attention layer is the last cross-attention layer in the feature enhancement model, output the key features as the compressed feature data; if the current cross-attention layer is not the last cross-attention layer in the feature enhancement model, based on the key features and the query feature data corresponding to the next cross-attention layer, fuse and generate the enhanced query data corresponding to the next cross-attention layer, and set the next cross-attention layer as the current cross-attention layer.

[0152] According to one or more embodiments of the present disclosure, the processing module is also used to: obtain target hyperparameters of the feature enhancement model; and determine the number of cross-attention layers in the feature enhancement model based on the target hyperparameters.

[0153] According to one or more embodiments of the present disclosure, the formatted feature data includes a key sequence and a value sequence. When the processing module processes the corresponding enhanced query data and the formatted feature data through the current cross-attention layer to obtain the key features corresponding to the current cross-attention layer, it is specifically used to: perform linear transformations on the enhanced query data, the key sequence and the value sequence, respectively, to obtain corresponding query vectors, key vectors and value vectors; obtain a scaling factor according to the vector dimension of the key vector; obtain an attention score matrix according to the product of the query vector and the key vector and the ratio of the scaling factor; process the attention score matrix using a normalized exponential function to obtain an attention weight matrix; and obtain the key features according to the product of the attention weight matrix and the value vector.

[0154] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0155] The memory stores computer-executable instructions;

[0156] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the feature data compression method described in the first aspect and various possible designs of the first aspect.

[0157] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the characteristic data compression method described in the first aspect and various possible designs of the first aspect is implemented.

[0158] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the characteristic data compression method described in the first aspect and various possible designs of the first aspect.

[0159] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0160] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0161] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A feature data compression method, characterized in that: include: Acquire an initial feature sequence of a target video, wherein the initial feature sequence represents video content features of the target video; According to the initial feature sequence, enhanced query data and formatted feature data are generated respectively, wherein the enhanced query data is constructed based on the global feature data, the global feature data is data after dimension reduction is performed on the time dimension of the initial feature sequence, and the formatted feature data is data after the initial feature sequence is formatted based on the target data format; The enhanced query data and the formatted feature data are processed separately by a feature enhancement model to obtain compressed feature data of the target video, wherein the feature enhancement model includes a plurality of cross-attention layers, and the plurality of cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate the compressed feature data.

2. The method according to claim 1, characterized in that The initial feature sequence includes a time dimension component, a space dimension component and a channel dimension component, wherein the time dimension component is used to characterize the playback time sequence of the target video, the space dimension component is used to characterize the image features of the video frames corresponding to each timestamp in the playback time sequence, and the channel dimension component is used to characterize the number of feature channels of the image features.

3. The method according to claim 2, characterized in that The step of generating enhanced query data according to the initial feature sequence comprises: Averaging the initial feature sequence on the time dimension component to obtain global feature data including only the space dimension component and the channel dimension component; The enhanced query data is obtained according to the global feature data.

4. The method according to claim 3, characterized in that The method further comprises: Creating an initial query feature according to the feature sizes of the spatial dimension component and the channel dimension component, wherein the initial query feature is a learnable feature matrix of all zeros; The step of obtaining enhanced query data according to the global feature data includes: Enhanced query data is obtained according to the sum of the initial query feature and the global feature data.

5. The method according to claim 2, characterized in that: Generating formatted feature data according to the initial feature sequence includes: The time dimension component and the space dimension component of the initial feature sequence are fused to obtain the time and space dimension component; The formatted feature data is generated according to the spatiotemporal dimension component and the channel dimension component.

6. The method according to claim 1, characterized in that The step of processing the enhanced query data and the formatted feature data respectively through the feature enhancement model to obtain compressed feature data of the target video includes: The following steps are executed in a loop: Processing the corresponding enhanced query data and the formatted feature data through the current cross attention layer to obtain the key features corresponding to the current cross attention layer; If the current cross-attention layer is the last cross-attention layer in the feature enhancement model, outputting the key feature as the compressed feature data; If the current cross-attention layer is not the last cross-attention layer in the feature enhancement model, the enhanced query data corresponding to the next cross-attention layer is generated based on the key features and the query feature data corresponding to the next cross-attention layer, and the next cross-attention layer is set as the current cross-attention layer.

7. The method according to claim 6, characterized in that The method further comprises: Obtaining target hyperparameters of the feature enhancement model; The number of cross-attention layers in the feature enhancement model is determined according to the target hyperparameters.

8. The method according to claim 6, characterized in that The formatted feature data includes a key sequence and a value sequence, and the corresponding enhanced query data and the formatted feature data are processed through the current cross-attention layer to obtain the key features corresponding to the current cross-attention layer, including: Performing linear transformation on the enhanced query data, the key sequence and the value sequence respectively to obtain corresponding query vectors, key vectors and value vectors; Obtaining a scaling factor according to the vector dimension of the key vector; Obtaining an attention score matrix according to a ratio of the product of the query vector and the key vector to the scaling factor; Processing the attention score matrix using a normalized exponential function to obtain an attention weight matrix; The key feature is obtained according to the product of the attention weight matrix and the value vector.

9. A characteristic data compression device, characterized in that: include: An acquisition module, used to acquire an initial feature sequence of a target video, wherein the initial feature sequence represents video content features of the target video; A generating module, configured to generate enhanced query data and formatted feature data according to the initial feature sequence, wherein the enhanced query data is constructed based on global feature data, the global feature data is data after dimension reduction is performed on the time dimension of the initial feature sequence, and the formatted feature data is data after the initial feature sequence is formatted based on a target data format; A processing module is used to process the enhanced query data and the formatted feature data respectively through a feature enhancement model to obtain compressed feature data of the target video, wherein the feature enhancement model includes a plurality of cross-attention layers, and the plurality of cross-attention layers are used to gradually extract key features in the formatted feature data based on the correlation between the enhanced query data and the formatted feature data to generate the compressed feature data.

10. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the feature data compression method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the feature data compression method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the feature data compression method according to any one of claims 1 to 8 is implemented.

Images