Data feature storage method and device

Abstract

The embodiment of the application provides a data feature storage and calculation method and device, the method comprises the following steps: obtaining video data from a plurality of different data source interfaces, and converting the video data into a preset task format to obtain a task stream containing meta information; performing frame extraction processing on the video data according to the task stream to obtain a video frame sequence; performing feature extraction on the video frame sequence to obtain a video feature vector, and storing the video feature vector and the meta information into a distributed storage module. By converting various types of input video data into a unified task format, then processing these tasks to obtain feature data, and persisting the feature data into a distributed storage system, the calculated feature data can be directly obtained for model training or reasoning, without repeated calculation. Moreover, the processing mode of the streaming task ensures the continuous flow of data in each processing stage, avoids intermediate waiting, and improves the overall system processing efficiency.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a data feature storage and calculation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large language models have gradually become a research hotspot. The introduction of video understanding capabilities enables these models to better understand complex visual information. The typical workflow of existing multimodal large language models for processing video data is as follows:

[0003] Model training phase: Videos are acquired from video data sources, preprocessed (including downloading, frame extraction, standardization, etc.), video features are extracted using a visual encoder, and then the features are input into the model for training.

[0004] Model inference stage: Receive the video and prompt words input by the user, perform the same preprocessing and feature extraction on the video, input the features and prompt words together into the model for inference, and finally return the inference result.

[0005] In existing technologies, video preprocessing and feature extraction are typically tightly coupled with model training or inference, requiring each business scenario's model to undergo video processing and feature extraction again. This approach presents several problems: the same video needs to be repeatedly preprocessed and feature extracted in different business scenarios, resulting in redundant consumption of CPU and GPU resources; furthermore, preprocessing (IO and CPU intensive) and feature extraction (GPU intensive) are bundled together, causing GPU resources to be idle while processing preprocessing tasks, reducing overall GPU utilization; and each business process performs video processing independently, increasing the burden on storage and network bandwidth, requiring each business team to maintain its own video processing workflow, thus increasing R&D and maintenance manpower costs.

[0006] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0007] This application provides a data feature storage and computing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems mentioned above.

[0008] One aspect of this application provides a data feature storage and computation method, the method comprising:

[0009] Video data is obtained from multiple different data source interfaces, and the video data is converted into a preset task format to obtain a task stream containing metadata.

[0010] The video data is processed by frame extraction according to the task flow to obtain a video frame sequence;

[0011] The video frame sequence is subjected to feature extraction to obtain a video feature vector, and the video feature vector and metadata are stored in a distributed storage module.

[0012] Optionally, after the step of converting the video data into a preset task format to obtain a task flow containing metadata, the method further includes:

[0013] The metadata corresponding to the video data is written into the write-ahead log module according to the task flow;

[0014] The status of each preprocessing task in the task flow is checked based on the unique key field in the metadata.

[0015] If the status of the preprocessing task is completed or the number of retries has been exceeded, an ACK confirmation message is generated.

[0016] Optionally, the method further includes:

[0017] When the status of the preprocessing task is "needs processing", the preprocessing task is passed to the task flow, and after the preprocessing task is completed, the preprocessing task is deleted from the write-ahead log module.

[0018] Optionally, the method further includes:

[0019] The task flow in the write-ahead log module is periodically scanned to identify timed-out tasks, and the timed-out tasks are retransmitted into the task flow for re-consumption.

[0020] Optionally, the step of extracting features from the video frame sequence to obtain a video feature vector includes:

[0021] Write the video frame sequence into a buffer queue;

[0022] The current video frame is sequentially retrieved from the buffer queue, and the current video frame is used to extract features to obtain a video feature vector.

[0023] Optionally, storing the video feature vector in the distributed storage module includes:

[0024] The video feature vector is serialized to obtain binary format feature data, and the binary format feature data is stored in the distributed storage module.

[0025] Optionally, the method further includes:

[0026] Receive model training tasks or inference prediction tasks for the video to be processed;

[0027] The system queries the distributed storage module to determine whether a target feature vector matching the video to be processed exists in the pre-set feature components.

[0028] If a matching target feature vector exists, then model training or inference prediction is performed based on the target feature vector;

[0029] If no matching target feature vector exists, an instant feature processing flow is triggered to extract the target feature vector corresponding to the video to be processed, and the target feature vector is stored in the distributed storage module.

[0030] Another aspect of this application provides a data feature storage and processing apparatus, the apparatus comprising:

[0031] The data source module is used to obtain video data from various different data source interfaces and convert the video data into a preset task format to obtain a task stream containing metadata.

[0032] The preprocessing module is used to perform frame extraction processing on the video data according to the task flow to obtain a video frame sequence;

[0033] The feature storage module is used to extract features from the video frame sequence to obtain video feature vectors, and store the video feature vectors and metadata in the distributed storage module.

[0034] Another aspect of this application provides a computer device, including:

[0035] At least one processor; and

[0036] A memory that is communicatively connected to the at least one processor;

[0037] Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data feature storage and calculation method as described above.

[0038] Another aspect of this application provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the data feature storage and calculation method described above.

[0039] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the data feature storage and calculation method described above.

[0040] The embodiments of this application employing the above technical solution may include the following advantages: by converting various types of input video data into a unified task format, processing these tasks to obtain feature data, and persisting the feature data to a distributed storage system, the calculated feature data can be directly obtained in the subsequent model training or inference prediction stage without repeated calculation. Moreover, this streaming task processing method ensures the continuous flow of data in each processing stage, avoids intermediate waiting, and improves the overall system processing efficiency. Attached Figure Description

[0041] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0042] Figure 1 This diagram schematically illustrates the operating environment of the data feature storage and computing method according to Embodiment 1 of this application;

[0043] Figure 2 A flowchart illustrating the data feature storage method according to Embodiment 1 of this application is shown in the schematic diagram.

[0044] Figure 3 A sequence diagram illustrating the workflow of a task manager is shown schematically.

[0045] Figure 4 This diagram illustrates a system framework for a data feature storage and computing scheme.

[0046] Figure 5 A block diagram of a data feature storage device according to Embodiment 2 of this application is schematically shown; and

[0047] Figure 6 A schematic diagram of the hardware architecture of a computer device according to Embodiment 3 of this application is shown. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0049] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0050] It should be noted that, in any stage of this application involving the collection, storage, use, transmission, and processing of data, each stage strictly adheres to the laws, regulations, industry standards, and regulatory requirements of the data source, usage location, and relevant countries and regions to ensure the legality and compliance of data activities. In the collection stage, the purpose, method, and scope of collection are clearly communicated to the data subject in a prominent manner. Collection is conducted only after obtaining the data subject's legal authorization, ensuring that the collection process follows the "minimum necessary" principle and does not exceed the scope of data collection. In the storage stage, storage periods are limited, and data is promptly deleted or anonymized / encrypted after the storage purpose is achieved. In the usage stage, a strict data security protection mechanism is implemented, using field-level desensitization technology and processing the original data according to preset desensitization rules. For different types of data, multiple desensitization strategies, such as data generalization, data anonymization, and data encryption, are employed to effectively mitigate the risk of sensitive information leakage and ensure that all data used is securely processed and desensitized, comprehensively protecting the rights and interests of data subjects and data security. In the transmission and processing stages, the confidentiality and security of data are ensured during transmission and processing.

[0051] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0052] First, a definition of the terminology used in this application is provided:

[0053] Ray Data Framework: A library within the AnyScale Ray ecosystem for building scalable data processing pipelines. It provides a set of high-level abstractions for loading, transforming, and computing large-scale datasets, supporting distributed and parallel data processing tasks.

[0054] Message queues are a communication mechanism used for asynchronous message delivery in distributed systems. Common message queues include Kafka and RabbitMQ, which allow producers to send messages to the queue, while consumers can independently receive and process messages from the queue. They are often used for processing streaming data.

[0055] Hive Table: A logical structure for organizing and storing data in Hive, a data warehouse tool built on top of Hadoop. Hive allows users to query large datasets stored in the Hadoop Distributed File System (HDFS) or other data storage systems using a SQL-like language (HQL), and is commonly used in batch processing scenarios.

[0056] Streaming data refers to data that is continuously generated and transmitted, typically requiring real-time or near real-time processing. Unlike batch data, streaming data is infinite in time.

[0057] Batch data refers to a collection of data collected and processed within a specific time period. Batch processing typically involves analyzing and manipulating the entire dataset.

[0058] DataSource Interface: An abstract programming interface used to unify the access methods of different data sources. By implementing this interface, the system can read data from various data storage systems (such as message queues, databases, file systems, etc.) in a consistent manner.

[0059] Hash: A function that maps an input of arbitrary length (also called a "message") to a fixed-length output (also called a "hash value," "message digest," or "message digest"). A good hash function should have the characteristics of uniqueness and uniform distribution, and is often used for data indexing, verification, and unique identification.

[0060] A distributed key-value store is a storage system that stores data as key-value pairs distributed across multiple servers. Common distributed key-value stores include Redis, Etcd, and Zookeeper, which typically offer high availability and scalability, and are suitable for storing metadata, caching, and configuration information.

[0061] Write-Ahead Log (WAL): A technique that records changes to data in a log file before the actual data modifications are performed. This helps in data recovery in the event of a system failure, ensuring data consistency and durability.

[0062] ACK (Acknowledgement): In communication protocols, an acknowledgment message sent by the receiver to the sender indicates that data has been successfully received. In message queues, consumers typically send an ACK to the message queue after successfully processing a message, informing it that the message can be safely deleted.

[0063] The preprocessor is responsible for performing preliminary processing and transformation of the raw data to facilitate subsequent operations such as feature extraction. In this patent, the preprocessor is primarily responsible for downloading and extracting frames from the video file.

[0064] Ray Actor: A basic unit of concurrent computation in Ray. An Actor is a stateful object that can execute tasks concurrently and maintain its own internal state. They are well-suited for building distributed stateful services and parallel processing pipelines.

[0065] The Decord library is a Python library for efficient video decoding, particularly suitable for processing and analyzing large-scale video data. It can quickly extract frames from video files with good performance.

[0066] I / O-bound: This refers to computer programs where the performance bottleneck lies primarily in input / output operations (such as disk read / write, network transmission, etc.), rather than CPU computation. Video downloading and frame extraction are typically I / O-bound tasks.

[0067] CPU-bound: This refers to computer programs whose performance bottleneck lies primarily in the computing power of the central processing unit (CPU), rather than in I / O operations. Some complex video processing or data conversion tasks may be CPU-bound.

[0068] Feature Extractor: Responsible for extracting feature vectors or feature maps from preprocessed data. In this patent, it uses a multimodal model to extract features from video frames.

[0069] Model weights: In machine learning models, these are the values ​​of the model parameters learned during training, used to predict or extract features from the input data. Loading model weights is a prerequisite for model inference or feature extraction.

[0070] GPU-bound: This refers to computer programs where the performance bottleneck lies primarily in the computing power of the graphics processing unit (GPU). It is particularly suitable for parallel computing-intensive tasks, such as training and inference of deep learning models. Feature extraction is typically a GPU-bound task.

[0071] Serialization is the process of converting the state of a data structure or object into a format that can be stored or transmitted (e.g., a byte stream). Deserialization is the process of restoring serialized data back to its original data structure or object.

[0072] A buffer queue is a queue used for temporary data storage, typically to decouple processing speed differences between different processing stages. In this patent, a buffer queue is used to cache preprocessed data during feature extraction to improve GPU utilization.

[0073] A distributed storage system is a storage system that distributes data across multiple computers to provide high availability, scalability, and fault tolerance. Common distributed storage systems include HDFS, S3, and Iceberg.

[0074] Iceberg: An open table format for analyzing large datasets. It provides features such as ACID transactions, schema evolution, and time travel, and is often used to build data lakes.

[0075] S3 (Simple Storage Service): In this article, it refers to any object storage service that implements the Amazon Web Services (AWS) Simple Storage Service (S3) API interface, which is commonly used to store various types of data.

[0076] Lazy computation is a computation strategy that delays the actual execution of computation until the result is truly needed. In the feature SDK of this patent, the getOrCompute interface embodies the concept of lazy computation, triggering computation only when the feature does not exist.

[0077] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below:

[0078] With the rapid development of artificial intelligence technology, multimodal large language models have gradually become a research hotspot. The introduction of video understanding capabilities enables these models to better understand complex visual information. The typical workflow of existing multimodal large language models for processing video data is as follows:

[0079] Model training phase: Videos are acquired from video data sources, preprocessed (including downloading, frame extraction, standardization, etc.), video features are extracted using a visual encoder, and then the features are input into the model for training.

[0080] Model inference stage: Receive the video and prompt words input by the user, perform the same preprocessing and feature extraction on the video, input the features and prompt words together into the model for inference, and finally return the inference result.

[0081] In existing technologies, video preprocessing and feature extraction are typically tightly coupled with model training or inference, requiring each business scenario's model to undergo video processing and feature extraction again. This approach presents the following problems:

[0082] Repetitive consumption of resources: The same video needs to be preprocessed and feature extracted repeatedly in different business scenarios, resulting in repeated consumption of CPU and GPU resources.

[0083] Inefficient computation: Preprocessing (IO and CPU intensive) and feature extraction (GPU intensive) are bundled together, causing GPU resources to be idle while processing preprocessing tasks, reducing overall GPU utilization.

[0084] Storage and bandwidth pressure: Each service processes video independently, increasing the burden on storage and network bandwidth.

[0085] High maintenance costs: Each business team needs to maintain the video processing workflow independently, which increases the manpower costs for research and development and maintenance.

[0086] Currently, the industry lacks a unified and efficient solution for calculating and storing multimodal large language model video features, which cannot effectively solve the above problems.

[0087] Based on the analysis of the above background technology, existing methods for processing video data using multimodal large language models mainly have the following drawbacks:

[0088] Low resource utilization: Because preprocessing (IO and CPU intensive) and feature extraction (GPU intensive) are bundled together, GPU resources are idle when performing preprocessing tasks, reducing GPU utilization.

[0089] Computational redundancy: The model training and inference processes in different business scenarios repeatedly perform preprocessing and feature extraction on the same video, resulting in a large amount of computational redundancy.

[0090] System throughput is limited: The integrated processing approach is difficult to optimize for different types of computing tasks, and it is impossible to set the batch size separately according to the characteristics of preprocessing and feature extraction tasks, thus limiting the overall throughput of the system.

[0091] High costs: Wasted resources due to redundant calculations increase computing costs, storage costs, bandwidth costs, and maintenance costs.

[0092] Poor scalability: The tightly coupled architecture struggles to adapt to the ever-increasing demands of video processing and cannot achieve efficient horizontal scaling.

[0093] Therefore, this application provides a technical solution for data feature storage and computation. In this solution, video data is obtained from various data source interfaces and converted into a preset task format to obtain a task flow containing metadata. Frame extraction is performed on the video data according to the task flow to obtain a video frame sequence. Feature extraction is performed on the video frame sequence to obtain video feature vectors, and the video feature vectors and metadata are stored in a distributed storage module. By converting various types of input video data into a unified task format, scheduling via a task manager is facilitated. Frame extraction is then performed on the video data to obtain a video frame sequence for feature extraction, and the feature data is persisted to a distributed storage system. This allows for direct retrieval of the calculated feature data during model training or inference prediction stages, eliminating the need for recalculation. Furthermore, this streaming task processing method ensures continuous data flow across processing stages, avoiding intermediate waiting and improving the overall system processing efficiency. Details are provided below.

[0094] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0095] like Figure 1 As shown in the diagram, the environment includes a service platform 2, a network 4, and a client 6, wherein:

[0096] Service platform 2 can consist of one or more computing devices. These computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as emulations of computer systems, operating systems, servers, etc. The computing devices can load virtual machines based on virtual images and / or other data that define specific software used for emulation (e.g., operating systems, dedicated applications, servers). As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.

[0097] Service platform 2 can be configured to communicate with clients 6, etc., via network 4. Network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar devices. Network 4 may include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, or combinations thereof, or wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0098] Service platform 2 can provide services such as storage, reading, writing, querying, and deletion, such as providing data feature storage and computing services for clients.

[0099] Client 6 can be an electronic device running operating systems such as Windows, Android™, or iOS, including smartphones, tablets, laptops, virtual reality devices, gaming devices, set-top boxes, in-vehicle terminals, and smart TVs. Based on these operating systems, various applications can run, such as applications for data feature storage and computation.

[0100] Client 6 can provide / configure user access pages, which can be used to control service platform 2 or upload objects, etc.

[0101] It should be noted that the above-mentioned equipment is exemplary, and the number and type of equipment can be adjusted in different scenarios or according to different needs.

[0102] The technical solution of this application will be described below through multiple embodiments, using service platform 2 as the implementing entity. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described herein.

[0103] Example 1

[0104] Figure 2 A flowchart illustrating the data feature storage method according to Embodiment 1 of this application is shown.

[0105] like Figure 2 As shown, the data feature storage and calculation method may include steps S202~S206, wherein:

[0106] Step S202: Obtain video data from multiple different data source interfaces, and convert the video data into a preset task format to obtain a task stream containing metadata;

[0107] In this embodiment, the distributed feature storage and computation system includes a data source module, DataSource. This DataSource module can obtain video data from various data source interfaces. As the data source head of the entire feature computation process, it is responsible for uniformly abstracting video data from different sources into a task flow that can be processed downstream. Its implementation is as follows:

[0108] Diverse Data Source Handling: For online or near-line scenarios, data typically originates from message queues (such as Kafka, RabbitMQ, etc.); for offline scenarios, data may be stored in distributed file systems (such as HDFS), data lakes (such as Hive, Iceberg, etc.), or object storage services (such as S3, etc.). Regardless of the underlying data source format and access method, the DataSource module implements a unified DataSource interface to stream the acquired video data into a task flow with a preset task format for subsequent feature calculation processes. The preset task format refers to a pre-defined processing task format. Specifically, the unified task format may include a video URL field and an ID field. The video URL field records the index address of the video data, and the ID field records the unique identifier of the video.

[0109] Task Meta-information Encapsulation: Each task output from the DataSource interface contains key metadata for subsequent processes such as task tracking, version management, and deduplication. This metadata includes at least the following fields:

[0110] namespace: Identifies the namespace of a business or application, used to distinguish the characteristic data of different businesses.

[0111] biz_id: Represents a unique identifier for business data, such as the ID of a video.

[0112] version: Represents the version number of the feature, used to manage different versions of the feature, facilitating model iteration and backtracking.

[0113] `unique_key`: Represents the unique key for a task. It is generated by combining the three metadata elements—namespace, biz_id, and version—and performing a hash operation. This globally unique `unique_key` plays a crucial role in subsequent task management and deduplication processes.

[0114] Step S204: Perform frame extraction processing on the video data according to the task flow to obtain a video frame sequence;

[0115] In this embodiment, the distributed feature storage and computation system includes a preprocessing component, whose main responsibility is to download video files and extract frames. Specifically, the preprocessing component uses a multi-threaded approach to download the video and extract video frames. As an example, the decord library can be used to achieve efficient video frame extraction. The decord library is often used in deep learning applications due to its efficiency in video decoding and frame extraction. The preprocessing process is typically I / O-intensive and CPU-intensive because it involves network transmission and video decoding computation.

[0116] To achieve efficient parallel processing, the preprocessing component is built on the Ray Actor model. The system launches a corresponding number of preprocessor Actor processes across the computing cluster, based on the configured number. Each preprocessor Actor consumes tasks from the output task stream of the Datasource component, downloads the specified video file, and then extracts video frames using the decor library. The extracted frame data is then appended to the task's context and streamed to the downstream feature extraction module. By using Ray Actor, preprocessing tasks can be executed in parallel across multiple nodes in the cluster, significantly improving overall processing speed and efficiency.

[0117] Step S206: Extract features from the video frame sequence to obtain video feature vectors, and store the video feature vectors and metadata in the distributed storage module.

[0118] In this embodiment, the distributed feature storage and computing system also includes a distributed feature extractor and a storage module. The distributed feature extractor, as the core computing component, is dedicated to efficiently extracting features from video frame sequences using a pre-trained multimodal model. This process is GPU-intensive, and its specific implementation is as follows:

[0119] First, to achieve high-throughput feature extraction, the Ray Actor framework can be used for parallel processing. The system launches multiple feature extractor Actor instances on GPU-equipped computing nodes based on the configured concurrency level. Each Actor is an independent process capable of executing feature extraction tasks concurrently, thus significantly improving overall processing power.

[0120] Each feature extractor (Actor) is responsible for loading the weights of a pre-trained multimodal model during initialization. Once loaded, the Actor continuously listens for and receives tasks streamed from the upstream distributed preprocessor. The context of each task includes the video frame sequence to be processed and related metadata.

[0121] Upon receiving a task containing a sequence of video frames, the feature extractor Actor loads this frame data into GPU memory and uses the loaded multimodal model to perform feature extraction calculations. The extracted feature vectors or feature map sequences serve as input data for the subsequent distributed storage module.

[0122] The distributed storage module is responsible for persistently storing the extracted video feature data and related metadata into the distributed storage system, and its design is as follows:

[0123] Storage backend selection: You can choose from a variety of distributed storage backends, such as data lakes (e.g., Iceberg, Delta Lake) and object storage services (e.g., S3, OSS), to meet different storage needs and cost considerations.

[0124] Data organization and management: Feature data is usually organized and managed according to certain rules in the storage system, such as partitioning or directory division based on metadata such as namespace, biz_id, and version, to facilitate subsequent retrieval and use.

[0125] Metadata storage: In addition to the feature data itself, related metadata (such as namespace, biz_id, version, feature dimension, storage path, etc.) will also be stored to facilitate the retrieval and management of the feature SDK.

[0126] In an optional embodiment of this application, after the step of streaming the video data to obtain a task stream containing metadata, the method further includes:

[0127] According to the task flow, the metadata corresponding to the video data is written to the write-ahead log module; the status of each preprocessing task in the task flow is checked according to the unique key field in the metadata; if the status of the preprocessing task is completed or the number of retries has been exceeded, an ACK confirmation message is generated.

[0128] In this embodiment, after a task is written to the WAL, the task manager can filter tasks based on the unique_key field in the metadata. If the task corresponding to the unique_key has been successfully processed, or its retry count exceeds the preset maximum threshold, it will be removed from the message queue (i.e., the task flow) and the WAL, and an ACK confirmation message will be generated.

[0129] In an optional embodiment of this application, the method further includes:

[0130] When the status of the preprocessing task is "needs processing", the preprocessing task is passed to the task flow, and after the preprocessing task is completed, the preprocessing task is deleted from the write-ahead log module.

[0131] In an optional embodiment of this application, the method further includes:

[0132] The task flow in the write-ahead log module is periodically scanned to identify timed-out tasks, and the timed-out tasks are retransmitted into the task flow for re-consumption.

[0133] To ensure the reliability and consistency of data processing, a task manager is introduced in this embodiment. This task manager is implemented based on a distributed key-value storage system (such as Redis, Etcd, Zookeeper, etc.) and is used to manage the lifecycle and status of tasks.

[0134] like Figure 3 A sequence diagram illustrating the workflow of a task manager is shown, from... Figure 3 As can be seen, the implementation of Task Manager mainly consists of the following three parts:

[0135] Write-Ahead Log (WAL): When the DataSource module reads a new task from the data source, before acknowledging (ACK) to the message queue, the task's metadata (including namespace, biz_id, version, and current timestamp) is first written to the Write-Ahead Log (WAL) maintained by the distributed key-value store system. This ensures that even if the DataSource module fails and restarts, unfinished tasks can be recovered from the WAL, avoiding data loss. Recovered tasks will have a higher consumption priority.

[0136] Task deduplication and filtering: After a task is written to the WAL, the task manager filters it based on its unique_key. If the task corresponding to the unique_key has already been successfully processed, or its retries exceed a preset maximum threshold, the task will not be passed to the downstream preprocessor and will be removed from the message queue and WAL, thus avoiding duplicate processing.

[0137] Task timeout retry: The Task Manager periodically scans the tasks in the WAL (Write-Ahead Log) to check if the processing time of a task exceeds a preset threshold. For timed-out tasks, the Task Manager rewrites them into the data stream so that the downstream preprocessor can consume them again, thus ensuring the task's eventual successful processing.

[0138] This design ensures data consistency and processing reliability in the face of various failure scenarios.

[0139] In an optional embodiment of this application, the step of extracting features from the video frame sequence to obtain a video feature vector includes:

[0140] Write the video frame sequence into a buffer queue;

[0141] The current video frame is sequentially retrieved from the buffer queue, and the current video frame is used to extract features to obtain a video feature vector.

[0142] In this embodiment, to address the GPU computing resource idleness issue (i.e., GPU bubble) caused by fluctuations in the upstream preprocessor's data output rate, each feature extractor actor implements a buffer queue. This mechanism decouples data acquisition from GPU computation into two independent execution threads.

[0143] Data acquisition thread: responsible for receiving tasks containing preprocessing results from upstream and putting video frame data (or pointers to them on the GPU) into a buffer queue.

[0144] GPU computing thread: continuously retrieves video frame data to be processed from the buffer queue and performs feature extraction calculations.

[0145] This producer-consumer model allows the GPU to directly retrieve input data for the next task from the buffer queue after completing feature extraction, without waiting for new data to be fully transmitted and ready from upstream. This asynchronous processing effectively hides data transmission latency, enabling the GPU to maintain high utilization and significantly improving feature extraction efficiency.

[0146] The entire feature extraction process is performed in a streaming manner, meaning that the feature extractor can begin working immediately once the preprocessor has generated enough video frame data, without waiting for the entire video preprocessing to complete. This near real-time processing method minimizes data processing latency and enables the entire distributed feature storage system to operate with high throughput and high resource utilization.

[0147] In an optional embodiment of this application, storing the video feature vector in a distributed storage module includes:

[0148] The video feature vector is serialized to obtain binary format feature data, and the binary format feature data is stored in the distributed storage module.

[0149] In this embodiment, after feature extraction is completed, to facilitate subsequent distributed storage and network transmission, the extracted feature data (usually a multidimensional floating-point tensor) is serialized into a compact binary format. The serialized binary data will be transmitted as part of the task context to the downstream distributed storage module in the form of a stream, so as to store the binary format feature data in the distributed storage module.

[0150] In one optional embodiment of this application, obtaining the video feature vector from the distributed storage module for model training or inference prediction during the model training or inference prediction stage includes:

[0151] The system receives a model training task or an inference prediction task for the video to be processed; it queries the distributed storage module to see if there is a target feature vector that matches the video to be processed through a preset feature component; if there is a matching target feature vector, it performs model training or inference prediction based on the target feature vector; if there is no matching target feature vector, it triggers an instant feature processing flow to extract the target feature vector corresponding to the video to be processed and stores the target feature vector in the distributed storage module.

[0152] In this embodiment, by setting up a feature SDK, the application layer can directly obtain feature vectors from the distributed storage module for model training or inference computation. This feature SDK acts as a bridge between the distributed feature storage system and the application layer, providing a unified feature access interface. The core implementation of the feature SDK is as follows:

[0153] Lazy computation interface (getOrCompute): First, it attempts to retrieve the computed features from distributed storage; if they do not exist, it triggers the computation process.

[0154] Batch Acquisition: Supports batch acquisition of multiple features, improving data acquisition efficiency.

[0155] Streaming access: Based on message queues, it provides streaming APIs and is suitable for scenarios that process large amounts of feature data.

[0156] Feature deserialization: Deserializes binary feature data into feature vectors for easier application.

[0157] The feature SDK allows for seamless integration of the training and inference frameworks into the system, skipping repetitive feature extraction steps and significantly reducing computational costs while improving efficiency.

[0158] By integrating this feature SDK, the inference and training frameworks can significantly save computational resources and development costs. Since feature extraction is a very time-consuming process, especially when dealing with large-scale video data, reusing pre-computed and stored features can significantly shorten model training and inference time and reduce the demand for computational resources.

[0159] The data feature storage and computation scheme in this embodiment can be implemented based on the Ray Data distributed computing framework, and the overall system architecture is as follows: Figure 4 As shown, Figure 4 The system comprises a data source module, a preprocessing module, a feature extraction module, a storage module, a feature SDK, and an application layer. The main steps of the data feature storage and processing flow through these modules include:

[0160] 1. The data source module converts various inputs into a unified task format and schedules them through the task manager;

[0161] 2. The preprocessing module receives the task, downloads the video and extracts frames, and then passes the video frame sequence to the feature extraction module.

[0162] 3. The feature extraction module uses a dual-thread design and a buffer queue mechanism to efficiently perform GPU feature calculations;

[0163] 4. The storage module persists the feature data to the distributed storage system;

[0164] 5. The application layer can directly obtain the calculated features through the getOrCompute interface of the feature SDK, without having to calculate them again.

[0165] This streaming processing method ensures the continuous flow of data at each processing stage, avoids intermediate waiting, and improves the overall system processing efficiency.

[0166] The data feature storage and computation scheme provided in this embodiment can perform unified, distributed, and efficient computation and storage of video features from online (near real-time) and offline (offline batch) scenarios, thereby significantly reducing the cost of model training and inference. By establishing a unified feature computation and storage layer, video feature data can be reused, improving system efficiency and reducing computational resource consumption.

[0167] This embodiment overcomes the shortcomings of the prior art through the following technical means:

[0168] Decoupling of computational processes: The process of processing videos using a multimodal large language model is decoupled into three independent stages: preprocessing, feature extraction, and inference / training, thereby achieving a reasonable allocation and efficient utilization of computational resources.

[0169] Heterogeneous Distributed Computing: Based on the Ray framework, a heterogeneous, streaming distributed computing architecture is implemented. Preprocessing and feature extraction are executed on CPU and GPU nodes respectively, and the number of computing nodes and batch size can be configured independently.

[0170] Streaming Computing Engine: A computing engine that supports both streaming and batch processing is designed, which can simultaneously process real-time / near-line video (message queue) and existing video (Hive table), achieving efficient utilization of computing resources.

[0171] Unified feature storage: A unified feature storage layer is constructed, which stores the extracted video features in binary form in a distributed storage system (such as Iceberg, S3) to achieve feature data persistence and reuse.

[0172] Feature SDK Interface: Provides a feature acquisition SDK, supports the getOrCompute interface, enabling inference and training frameworks to easily obtain calculated feature data and avoid redundant calculations.

[0173] Adaptive resource scheduling: Based on task characteristics and resource usage, adaptive scheduling of computing resources is achieved to maximize resource utilization and computing efficiency.

[0174] Example 2

[0175] Figure 5 A block diagram schematically illustrates a data feature storage and computing device according to Embodiment 2 of this application. This device can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of this application. The program module referred to in this embodiment is a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the function of each program module in this embodiment. For example... Figure 5 As shown, the data feature storage and processing device 500 may include: a data source module 501, a preprocessing module 502, and a feature storage module 503, wherein:

[0176] Data source module 501 is used to obtain video data from various different data source interfaces and convert the video data into a preset task format to obtain a task stream containing metadata.

[0177] Preprocessing module 502 is used to perform frame extraction processing on the video data according to the task flow to obtain a video frame sequence;

[0178] The feature storage module 503 is used to extract features from the video frame sequence to obtain video feature vectors, and store the video feature vectors and metadata in the distributed storage module.

[0179] In an optional embodiment of this application, the apparatus further includes:

[0180] The task management module is used to write the metadata corresponding to the video data into the write-ahead log module according to the task flow;

[0181] The status of each preprocessing task in the task flow is checked based on the unique key field in the metadata.

[0182] If the status of the preprocessing task is completed or the number of retries has been exceeded, an ACK confirmation message is generated.

[0183] In an optional embodiment of this application, the task management module is further configured to:

[0184] When the status of the preprocessing task is "needs processing", the preprocessing task is passed to the task flow, and after the preprocessing task is completed, the preprocessing task is deleted from the write-ahead log module.

[0185] In an optional embodiment of this application, the task management module is further configured to:

[0186] The task flow in the write-ahead log module is periodically scanned to identify timed-out tasks, and the timed-out tasks are retransmitted into the task flow for re-consumption.

[0187] In an optional embodiment of this application, the feature storage module 503 includes:

[0188] The feature extraction submodule is used to write the video frame sequence into a buffer queue; sequentially retrieve the current video frame from the buffer queue, and perform feature extraction on the current video frame to obtain a video feature vector.

[0189] In an optional embodiment of this application, the feature storage module 503 includes:

[0190] The feature storage submodule is used to serialize the video feature vector to obtain binary format feature data, and store the binary format feature data in the distributed storage module.

[0191] In an optional embodiment of this application, the apparatus further includes:

[0192] The feature acquisition module is used to receive model training tasks or inference prediction tasks for the video to be processed; query the distributed storage module through a preset feature component to see if there is a target feature vector matching the video to be processed; if there is a matching target feature vector, then model training or inference prediction is performed based on the target feature vector; if there is no matching target feature vector, then an instant feature processing flow is triggered to extract the target feature vector corresponding to the video to be processed, and the target feature vector is stored in the distributed storage module.

[0193] Example 3

[0194] Figure 6This illustration schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a data feature storage and computing method according to Embodiment 3 of this application. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual device, workbench, digital assistant, set-top box, robot, etc. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 6 As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus. Wherein:

[0195] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for data feature calculation methods. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0196] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0197] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0198] It should be pointed out that, Figure 6 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0199] In this embodiment, the data feature storage and calculation method stored in memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of this application.

[0200] Example 4

[0201] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the data feature calculation method in the embodiments.

[0202] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the data feature storage method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0203] Example 5

[0204] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above embodiments.

[0205] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0206] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A data feature storage and computation method, characterized in that, The method includes: Video data is obtained from multiple different data source interfaces, and the video data is converted into a preset task format to obtain a task stream containing metadata. The video data is processed by frame extraction according to the task flow to obtain a video frame sequence; The video frame sequence is subjected to feature extraction to obtain a video feature vector, and the video feature vector and metadata are stored in a distributed storage module; The method further includes: Receive model training tasks or inference prediction tasks for the video to be processed; The system queries the distributed storage module to determine whether a target feature vector matching the video to be processed exists in the pre-set feature components. If a matching target feature vector exists, then model training or inference prediction is performed based on the target feature vector; If no matching target feature vector exists, an instant feature processing flow is triggered to extract the target feature vector corresponding to the video to be processed, and the target feature vector is stored in the distributed storage module.

2. The data feature storage and computation method according to claim 1, characterized in that, After the step of converting the video data into a preset task format to obtain a task stream containing metadata, the method further includes: The metadata corresponding to the video data is written into the write-ahead log module according to the task flow; The status of each preprocessing task in the task flow is checked based on the unique key field in the metadata. If the status of the preprocessing task is completed or the number of retries has been exceeded, an ACK confirmation message is generated.

3. The data feature storage and computation method according to claim 2, characterized in that, The method further includes: When the status of the preprocessing task is "needs processing", the preprocessing task is passed to the task flow, and after the preprocessing task is completed, the preprocessing task is deleted from the write-ahead log module.

4. The data feature storage and computation method according to claim 2, characterized in that, The method further includes: The task flow in the write-ahead log module is periodically scanned to identify timed-out tasks, and the timed-out tasks are retransmitted into the task flow for re-consumption.

5. The data feature storage and computation method according to claim 1, characterized in that, The step of extracting video feature vectors from the video frame sequence includes: Write the video frame sequence into a buffer queue; The current video frame is sequentially retrieved from the buffer queue, and the current video frame is used to extract features to obtain a video feature vector.

6. The data feature storage and computation method according to claim 1, characterized in that, The step of storing the video feature vector into the distributed storage module includes: The video feature vector is serialized to obtain binary format feature data, and the binary format feature data is stored in the distributed storage module.

7. A data feature storage and processing device, characterized in that, The device includes: The data source module is used to obtain video data from various different data source interfaces and convert the video data into a preset task format to obtain a task stream containing metadata. The preprocessing module is used to perform frame extraction processing on the video data according to the task flow to obtain a video frame sequence; The feature storage module is used to extract features from the video frame sequence to obtain video feature vectors, and store the video feature vectors and metadata into the distributed storage module; The feature acquisition module is used for: Receive model training tasks or inference prediction tasks for the video to be processed; The system queries the distributed storage module to determine whether a target feature vector matching the video to be processed exists in the pre-set feature components. If a matching target feature vector exists, then model training or inference prediction is performed based on the target feature vector; If no matching target feature vector exists, an instant feature processing flow is triggered to extract the target feature vector corresponding to the video to be processed, and the target feature vector is stored in the distributed storage module.

8. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data feature storage and calculation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the data feature storage and calculation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the data feature storage and calculation method as described in claims 1 to 6.

Images