Data feature storage and calculation method and device

Abstract

The embodiment of the invention provides a data feature storage and calculation method and device, and the method comprises the steps: obtaining video data from a plurality of different data source interfaces, and converting the video data into a preset task format to obtain a task flow containing meta-information; performing frame extraction processing on the video data according to the task flow to obtain a video frame sequence; and performing feature extraction on the video frame sequence to obtain a video feature vector, and storing the video feature vector and meta information in a distributed storage module. Various types of input video data are converted into a unified task format, then the tasks are processed to obtain the feature data, and the feature data are persisted into the distributed storage system, so that the calculated feature data can be directly obtained to be used for model training or reasoning, repeated calculation is not needed, and the efficiency of model training or reasoning is improved. Moreover, the streaming task processing mode ensures continuous flow of the data in each processing stage, avoids intermediate waiting, and improves the processing efficiency of the whole system.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of Internet, and in particular, to a data feature storage and calculation method and device, computer equipment, computer readable storage medium, and computer program product. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, multi-modal large language models have gradually become a research hotspot, and the introduction of video understanding capabilities enables the model to better understand complex visual information. The typical process of existing multi-modal large language models for processing video data is as follows: Model training phase: Obtain video from a video data source, perform preprocessing (including downloading, frame extraction, standardization, etc.), extract video features using a visual encoder, and then input the features into the model for training.

[0003] Model inference phase: Receive a user-input video and prompt word, perform the same preprocessing and feature extraction on the video, input the features and the prompt word into the model for inference, and finally return the inference result.

[0004] In the prior art, the video preprocessing and feature extraction process is usually tightly coupled with model training or inference, and each model in a business scenario needs to reprocess video and extract features. This processing method has the following problems: the same video needs to be repeatedly preprocessed and feature-extracted in different business scenarios, causing repeated consumption of CPU and GPU resources; preprocessing (IO and CPU intensive) and feature extraction (GPU intensive) are bundled together, causing GPU resources to be idle when processing preprocessing tasks, reducing overall GPU utilization; each business independently processes video, increasing the burden on storage and network bandwidth, and each business team needs to independently maintain the video processing flow, increasing the human cost of research and development and maintenance.

[0005] It should be noted that the above content is not necessarily prior art and does not limit the patent protection scope of the present application. SUMMARY

[0006] Embodiments of the present application provide a data feature storage and calculation method, device, computer equipment, computer readable storage medium, and computer program product to solve or alleviate one or more technical problems raised above.

[0007] One aspect of an embodiment of the present application provides a data feature storage and calculation method, which includes: Obtaining video data from a plurality of different data source interfaces, and converting the video data into a preset task format to obtain a task stream containing meta information; Frame extraction processing the video data according to the task stream to obtain a video frame sequence; performing feature extraction on the video frame sequence to obtain a video feature vector, and storing the video feature vector and the meta information into a distributed storage module.

[0008] Optionally, after the step of converting the video data into a preset task format to obtain a task stream containing meta information, the method further comprises: writing the meta information corresponding to the video data into a prewrite log module according to the task stream; checking the state of each preprocessing task in the task stream according to the unique key field in the meta information; generating an ACK confirmation message in the case that the state of the preprocessing task is completed or exceeds the number of retries.

[0009] Optionally, the method further comprises: in the case that the state of the preprocessing task is to be processed, passing the preprocessing task to the task stream, and deleting the preprocessing task from the prewrite log module after the preprocessing task is processed.

[0010] Optionally, the method further comprises: periodically scanning the task stream in the prewrite log module to determine a timeout task, and re-passing the timeout task to the task stream to re-consume the timeout task.

[0011] Optionally, the performing feature extraction on the video frame sequence to obtain a video feature vector comprises: writing the video frame sequence into a buffer queue; sequentially obtaining a current video frame from the buffer queue, and performing feature extraction on the current video frame to obtain a video feature vector.

[0012] Optionally, the storing the video feature vector into a distributed storage module comprises: performing serialization processing on the video feature vector to obtain binary format feature data, and storing the binary format feature data into the distributed storage module.

[0013] Optionally, the method further comprises: receiving a model training task or an inference prediction task for a to-be-processed video; querying whether there is a target feature vector matching the to-be-processed video in the distributed storage module through a preset feature component; in the case that there is a matching target feature vector, then performing model training or inference prediction according to the target feature vector; In the absence of a matching target feature vector, an instant feature processing flow is triggered to extract a target feature vector corresponding to the video to be processed, and the target feature vector is stored in the distributed storage module.

[0014] Another aspect of the embodiments of the present application provides a data feature storage and calculation device, which comprises: A data source module is configured to obtain video data from a plurality of different data source interfaces, and convert the video data into a preset task format to obtain a task stream containing meta information. A preprocessing module is configured to perform frame extraction processing on the video data according to the task stream to obtain a video frame sequence. A feature storage module is configured to perform feature extraction on the video frame sequence to obtain a video feature vector, and store the video feature vector and the meta information in a distributed storage module.

[0015] Another aspect of the embodiments of the present application provides a computer device, which comprises: at least one processor; and a memory in communication connection with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data feature storage and calculation method as described above.

[0016] Another aspect of the embodiments of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the data feature storage and calculation method as described above.

[0017] Another aspect of the embodiments of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the data feature storage and calculation method as described above.

[0018] The embodiments of the present application can include the following advantages by adopting the above technical solutions: by converting various input video data into a unified task format, then processing these tasks to obtain feature data, and persisting the feature data in a distributed storage system, in the subsequent model training stage or inference prediction stage, the calculated feature data can be directly obtained without repeated calculation, and the processing mode of the stream task ensures the continuous flow of data in each processing stage, avoids intermediate waiting, and improves the processing efficiency of the overall system. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are exemplary only and not limiting of the scope of the appended claims. In all

[0020] Figure 1 Fig. 1 schematically shows a running environment diagram of a data feature storage and calculation method according to Embodiment One of the present application; Figure 2 Fig. 2 schematically shows a flowchart of a data feature storage and calculation method according to Embodiment One of the present application; Figure 3 Fig. 3 schematically shows a timing diagram of a task manager workflow; Figure 4 Fig. 4 schematically shows a system framework diagram of a data feature storage and calculation scheme; Figure 5 Fig. 5 schematically shows a block diagram of a data feature storage and calculation device according to Embodiment Two of the present application; and Figure 6 Fig. 6 schematically shows a hardware architecture diagram of a computer device according to Embodiment Three of the present application. DETAILED DESCRIPTION

[0021] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. used in the embodiments of the present application are only for the purpose of description and should not be understood as indicating or implying the relative importance of the technical features indicated or the number of the technical features indicated. Therefore, the features with "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection claimed by the present application.

[0023] It should be noted that in the present application, if the collection, storage, use, transmission and processing of data are involved, each link of the data shall strictly follow the laws, regulations, industry standards and regulatory requirements of the data source, use location and relevant countries and regions to ensure the legality and compliance of data activities. In the collection link, the data subject is informed of the collection purpose, method and scope in a conspicuous manner, and the collection is carried out after obtaining the legal authorization of the data subject, ensuring that the collection process follows the "minimum necessary" principle and does not collect data beyond the scope. In the storage link, the storage period is limited, and the data is deleted or anonymized and encrypted in a timely manner after achieving the storage purpose. In the use link, strict data security protection mechanisms will be implemented, and field-level desensitization technology will be used to process the original data according to the preset desensitization rules. For different types of data, various desensitization strategies such as data generalization, data anonymization and data encryption are adopted to effectively avoid the risk of sensitive information leakage and ensure that the data used is desensitized data after security processing, thereby fully protecting the rights and interests of data subjects and data security. In the transmission and processing link, the confidentiality and security of data in the transmission and processing process are ensured.

[0024] In the description of the present application, it should be understood that the numerical reference before the step does not indicate the order of execution of the steps before and after the step, but is only used to facilitate the description of the present application and to distinguish each step, and therefore cannot be understood as a limitation on the present application.

[0025] First, provide explanations of the terms involved in the present application: Ray Data Framework: A library in the AnyScale Ray ecosystem for building scalable data processing pipelines. It provides a set of high-level abstractions for loading, transforming, and computing large-scale datasets, supporting distributed and parallel data processing tasks.

[0026] Message Queue: A communication mechanism used for asynchronous message delivery in distributed systems. Common message queues include Kafka, RabbitMQ, etc., which allow producers to send messages to queues, while consumers can independently receive and process messages from queues, commonly used for processing streaming data.

[0027] Hive Table: A logical structure for organizing and storing data in Hive, a data warehouse tool built on Hadoop. Hive allows users to query large-scale datasets stored in Hadoop Distributed File System (HDFS) or other data storage systems using SQL-like language (HQL), commonly used in batch processing scenarios.

[0028] Streaming Data: Refers to data that is continuously generated and transmitted, often requiring real-time or near-real-time processing. Unlike batch data, streaming data is infinite in time.

[0029] Batch Data: Refers to a collection of data that is collected and processed within a specific time period. Batch processing typically involves analyzing and operating on the entire data set.

[0030] DataSource Interface: An abstract programming interface that unifies the access methods of different data sources. By implementing this interface, systems can read data from various data storage systems (such as message queues, databases, file systems, etc.) in a consistent manner.

[0031] Hash: A function that maps an input of arbitrary length (also known as "message") to a fixed-length output (also known as "hash value", "digest" or "message digest"). Good hash functions should have the properties of uniqueness and uniform distribution, commonly used for data indexing, checksums, and unique identification.

[0032] Distributed Key-Value Store: A storage system that stores data as key-value pairs and distributes them across multiple servers. Common distributed key-value systems include Redis, Etcd, Zookeeper, etc., which usually provide high availability and scalability, suitable for storing metadata, caching and configuration information, etc.

[0033] Write-Ahead Log (WAL): A technique that records modification operations to a log file before performing actual data modification. This helps to recover data in case of system failure, ensuring data consistency and persistence.

[0034] ACK (Acknowledgement): In communication protocols, the receiving party sends an acknowledgment message to the sending party, indicating that the data has been successfully received. In message queues, consumers usually send ACK to the message queue after successfully processing the message, indicating that the message can be safely deleted.

[0035] Preprocessor: Responsible for the initial processing and conversion of raw data to facilitate subsequent feature extraction and other operations. In this patent, the preprocessor is mainly responsible for downloading and frame extraction of video files.

[0036] Ray Actor: A basic concurrent computing unit in Ray. Actors are stateful objects that can execute tasks concurrently and maintain their own internal state. They are well-suited for building distributed stateful services and parallel processing pipelines.

[0037] Decord Library: A Python library for efficient video decoding, particularly suitable for large-scale video data processing and analysis. It can quickly extract frames from video files with good performance.

[0038] I / O-bound: Refers to a computer program whose performance bottleneck is primarily in input / output operations (such as disk reading and writing, network transmission, etc.), rather than CPU computation. Video downloading and frame extraction are usually I / O-bound tasks.

[0039] CPU-bound: Refers to a computer program whose performance bottleneck is primarily in the central processing unit (CPU) computing power, rather than I / O operations. Some complex video processing or data conversion tasks may be CPU-bound.

[0040] Feature Extractor: Responsible for extracting feature vectors or feature maps from pre-processed data. In this patent, it uses a multi-modal model to extract features from video frames.

[0041] Model Weights: In machine learning models, the values of model parameters learned during the training process, used for prediction or feature extraction on input data. Loading model weights is a prerequisite for model inference or feature extraction.

[0042] GPU-bound: Refers to a computer program whose performance bottleneck is primarily in the graphics processing unit (GPU) computing power, especially suitable for parallel computing-intensive tasks such as deep learning model training and inference. Feature extraction is usually a GPU-bound task.

[0043] Serialization: The process of converting the state of a data structure or object into a format that can be stored or transmitted (for example, a byte stream). Deserialization is the process of restoring the serialized data to the original data structure or object.

[0044] Buffer Queue: A queue used to temporarily store data, usually used to decouple the processing speed difference between different processing stages. In this patent, the buffer queue is used to cache pre-processed data during feature extraction to improve GPU utilization.

[0045] Distributed Storage System: A storage system that distributes data across multiple computers to provide high availability, scalability, and fault tolerance. Common distributed storage systems include HDFS, S3, Iceberg, etc.

[0046] Iceberg: An open table format for large-scale analytical datasets. It provides ACID transactions, schema evolution, time travel, etc., and is commonly used to build data lakes.

[0047] S3 (Simple Storage Service): In this context, it refers to any object storage service that implements the Amazon Web Services (AWS) Simple Storage Service (S3) API interface, commonly used to store various types of data.

[0048] Lazy Computation: A computing strategy that delays the actual execution of computation until the result is truly needed. In the feature SDK of this patent, the getOrCompute interface embodies the idea of lazy computation, triggering computation only when the feature does not exist.

[0049] Secondly, to facilitate the understanding of the technical solutions provided by the technical personnel in the field, the related technologies are described as follows: With the rapid development of artificial intelligence technology, multi-modal large language models have gradually become a research hotspot. The introduction of video understanding capabilities enables the model to better understand complex visual information. The typical process of existing multi-modal large language models processing video data is as follows: Model training phase: Obtain video data from video data sources, perform preprocessing (including downloading, frame extraction, standardization, etc.), use a visual encoder to extract video features, and then input the features into the model for training.

[0050] Model inference phase: Receive user input video and prompt words, perform the same preprocessing and feature extraction on the video, input the features and prompt words into the model for inference, and finally return the inference result.

[0051] In existing technologies, the video preprocessing and feature extraction process is usually tightly coupled with model training or inference. Each model in different business scenarios needs to reprocess video and extract features. This processing method has the following problems: Resource duplication: The same video needs to be preprocessed and feature extracted repeatedly in different business scenarios, causing repeated consumption of CPU and GPU resources.

[0052] Low computational efficiency: Preprocessing (IO and CPU-intensive) and feature extraction (GPU-intensive) are bundled together, causing GPU resources to be idle when processing preprocessing tasks, reducing overall GPU utilization.

[0053] Storage and bandwidth pressure: Each business independently processes video, increasing the burden on storage and network bandwidth.

[0054] High maintenance cost: Each business team needs to independently maintain the video processing process, increasing the cost of research and development and maintenance manpower.

[0055] Currently, there is a lack of a unified and efficient multi-modal large language model video feature calculation and storage solution in the industry, which cannot effectively solve the above problems.

[0056] Based on the analysis of the above background technology, the existing multi-modal large language model processing video data method mainly has the following shortcomings: Low resource utilization: Since preprocessing (IO and CPU-intensive) and feature extraction (GPU-intensive) are bundled together, GPU resources are idle when performing preprocessing tasks, reducing GPU utilization.

[0057] Computational redundancy: Model training and inference processes in different business scenarios repeatedly preprocess and extract features from the same video, causing a lot of computational redundancy.

[0058] Limited system throughput: The integrated processing method is difficult to optimize for different types of computing tasks, and cannot set batch sizes according to the characteristics of preprocessing and feature extraction tasks, limiting the overall throughput of the system.

[0059] High cost: Resource waste caused by repeated calculations increases the cost of computing, storage, bandwidth, and maintenance.

[0060] Poor scalability: The tightly coupled architecture is difficult to adapt to growing video processing needs and cannot achieve efficient horizontal expansion.

[0061] To this end, an embodiment of the present application provides a technical solution of data feature storage and calculation. In the technical solution, video data is obtained from a plurality of different data source interfaces, and the video data is converted into a preset task format to obtain a task stream containing meta information; the video data is frame extracted according to the task stream to obtain a video frame sequence; the video frame sequence is feature extracted to obtain a video feature vector, and the video feature vector and the meta information are stored in a distributed storage module. By converting various input video data into a unified task format, it is convenient to schedule through a task manager, then frame extract the video data to obtain a video frame sequence for feature extraction, and persist the feature data to a distributed storage system, so that in the subsequent model training stage or inference prediction stage, the calculated feature data can be directly obtained without repeated calculation, and the processing mode of the stream task ensures the continuous flow of data in each processing stage, avoids intermediate waiting, and improves the processing efficiency of the overall system. See the following.

[0062] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0063] As shown in Figure 1 The environment diagram includes a service platform 2, a network 4, and a client 6, wherein: The service platform 2 can be composed of a single or multiple computing devices. The multiple computing devices can include virtualized computing instances. The virtualized computing instances can include virtual machines, such as emulations of computer systems, operating systems, servers, and the like. The computing devices can load the virtual machines based on virtual images and / or other data defining specific software (e.g., operating systems, specialized applications, servers) for emulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.

[0064] The service platform 2 can be configured to communicate with the client 6 and the like through the network 4. The network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 can include physical links, such as coaxial cable links, twisted-pair cable links, fiber-optic links, combinations thereof, and the like, or wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.

[0065] The service platform 2 can provide storage, reading, writing, querying, deletion, and the like services, such as providing data feature storage and calculation services for the client.

[0066] The client 6 can be an electronic device running an operating system such as Windows, Android™, or iOS, such as a smartphone, a tablet device, a laptop computer, a virtual reality device, a game device, a set-top box, a vehicle terminal, a smart television. Based on the above operating system, various application programs can be run, such as an application program for data feature storage and calculation.

[0067] The client 6 can provide / configure a user access page for manipulating the service platform 2 or uploading objects, etc.

[0068] It should be noted that the above devices are exemplary, and the number and types of devices can be adjusted in different scenarios or according to different needs.

[0069] The following describes the technical solutions of the present application through a plurality of embodiments with the service platform 2 as the execution subject. It should be understood that these embodiments can be implemented in a variety of different forms, and should not be interpreted as being limited to the embodiments set forth herein.

[0070] Embodiment One Figure 2 A flowchart of a data feature storage and calculation method according to Embodiment One of the present application is schematically shown.

[0071] As shown in Figure 2 The data feature storage and calculation method can include steps S202-S206, wherein: Step S202, video data is obtained from a plurality of different data source interfaces, and the video data is converted into a preset task format to obtain a task stream containing meta information; In this embodiment, the distributed feature storage and calculation system includes a data source module DataSource, which can obtain video data from a plurality of different data source interfaces through the DataSource module. The DataSource module serves as the data source of the entire feature calculation process and is responsible for uniformly abstracting video data of different sources into a task stream for downstream processing. The implementation manner is as follows: Data source diversity processing: For online or nearline scenarios, data is usually sourced from message queues (such as Kafka, RabbitMQ, etc.); for offline scenarios, data may be stored in distributed file systems (such as HDFS), data lakes (such as Hive, Iceberg, etc.), or object storage services (such as S3, etc.). Regardless of the format and access method of the underlying data source, the DataSource module outputs the video data stream obtained in the preset task format task stream for subsequent feature calculation process consumption by implementing a unified DataSource interface. Among them, the preset task format refers to the pre-set processing task format, specifically, the unified task format can include a video URL field and an ID field, the video URL field is used to record the index address of the video data, and the ID field is used to record the unique identification code of the video. Task meta information encapsulation: Each task output from the DataSource interface contains key meta information for task tracking, version management, and deduplication in subsequent processes. Among them, these meta information at least includes the following fields: namespace: Identifies the namespace of the business or application, used to distinguish the feature data of different businesses.

[0072] biz_id: Represents the unique identifier of the business data, for example, the ID of the video.

[0073] version: Represents the version number of the feature, used to manage different versions of the feature, facilitating model iteration and backtracking.

[0074] unique_key: Represents the unique key of the task, which is generated by combining and hashing the namespace, biz_id, and version meta information. The unique_key plays a key role in subsequent task management and deduplication.

[0075] Step S204, frame extraction processing is performed on the video data according to the task stream to obtain a video frame sequence; In this embodiment, the distributed feature storage and computing system includes a preprocessing component, which is mainly responsible for downloading video files and performing frame extraction. Specifically, the preprocessing component downloads videos and performs video frame extraction in a multi-threaded manner. As an example, the decord library can be used to achieve efficient video frame extraction. The decord library is often used in deep learning applications due to its efficiency in video decoding and frame extraction. The preprocessing process is usually I / O intensive and CPU intensive, as it involves network transmission and video decoding calculations.

[0076] To achieve efficient parallel processing, the preprocessing component is built based on the Ray Actor model. The system will start the corresponding number of preprocessor Actor processes in the computing cluster according to the configured number. Each preprocessor Actor will consume tasks from the output task stream of the Datasource component, download the video file specified in the task, and then extract video frames using the decord library. The extracted frame data is then attached to the context of the task and passed to the downstream feature extraction module in a streaming manner. By using Ray Actor, the preprocessing task can be executed in parallel on multiple nodes in the cluster, significantly improving the overall processing speed and efficiency.

[0077] Step S206, feature extraction is performed on the video frame sequence to obtain a video feature vector, and the video feature vector and meta information are stored in a distributed storage module.

[0078] In this embodiment, the distributed feature storage and computing system further includes a distributed feature extractor and a storage module. The distributed feature extractor, as a core computing component, is dedicated to efficiently extracting features from video frame sequences using pre-trained multi-modal models. This process is a GPU-intensive operation, and its specific implementation is as follows: First, to achieve high-throughput feature extraction, Ray Actor framework can be used for parallel processing. The system starts multiple feature extractor Actor instances on computing nodes equipped with GPUs according to the configured concurrency. Each Actor is an independent process that can concurrently execute feature extraction tasks, significantly improving overall processing capacity.

[0079] Each feature extractor Actor is responsible for loading the weights of the pre-trained multi-modal model during initialization. After loading is complete, the Actor will continuously listen and receive tasks passed in a streaming manner from the upstream distributed preprocessor. The context of each task contains the video frame sequence to be processed and related metadata.

[0080] After receiving a task containing a video frame sequence, the feature extractor Actor loads these frame data into GPU memory and performs feature extraction computation using the loaded multi-modal model. The extracted feature vector or feature map sequence is the input data for the subsequent distributed storage module.

[0081] The distributed storage module is responsible for persistently storing the extracted video feature data and related metadata in the distributed storage system, and its design is as follows: Storage backend selection: a variety of distributed storage backends can be selected, such as data lakes (e.g. Iceberg, DeltaLake), object storage services (e.g. S3, OSS), etc., to meet different storage requirements and cost considerations.

[0082] Data organization and management: Feature data is usually organized and managed in a storage system according to certain rules, such as partitioning or directory division according to namespace, biz_id, version, and other meta information, to facilitate subsequent retrieval and use.

[0083] Metadata storage: In addition to the feature data itself, related meta information (such as namespace, biz_id, version, feature dimension, storage path, etc.) is also stored to facilitate retrieval and management by the feature SDK.

[0084] In an optional embodiment of the present application, after the step of streaming the video data to obtain a task stream containing meta information, the method further comprises: writing the meta information corresponding to the video data into a pre-write log module according to the task stream; checking the state of each preprocessing task in the task stream according to the unique key field in the meta information; and generating an ACK confirmation message in the case that the state of the preprocessing task is completed or exceeds the number of retries.

[0085] In this embodiment, after the task is written into the WAL, the task manager can filter the task based on the unique key field unique_key in the meta information. If the task corresponding to the unique_key has been successfully processed or the number of retries exceeds the preset maximum threshold, it will be removed from the message queue (i.e., the task stream) and the WAL, and an ACK confirmation message will be generated.

[0086] In an optional embodiment of the present application, the method further comprises: in the case that the state of the preprocessing task needs to be processed, passing the preprocessing task to the task stream, and deleting the preprocessing task from the pre-write log module after the preprocessing task is processed.

[0087] In an optional embodiment of the present application, the method further comprises: periodically scanning the task stream in the pre-write log module to determine timeout tasks, and re-passing the timeout tasks to the task stream to re-consume the timeout tasks.

[0088] To ensure the reliability and consistency of data processing, a task manager is introduced in this embodiment, which is implemented based on a distributed key-value storage system (such as Redis, Etcd, Zookeeper, etc.) and is used to manage the life cycle and state of tasks.

[0089] As Figure 3A timing diagram showing the workflow of the task manager is shown from Figure 3 As can be seen, the implementation of the task manager mainly includes the following three parts: Write-Ahead Log (WAL): When the DataSource module reads a new task from the data source, the meta information of the task (including namespace, biz_id, version, and current timestamp) will first be written into the Write-Ahead Log (WAL) maintained by the distributed key-value storage system before responding (ACK) to the message queue. This ensures that even if the DataSource module fails and restarts, the unfinished task can be recovered from the WAL to avoid data loss. The recovered task will have a higher consumption priority.

[0090] Task deduplication and filtering: After the task is written into the WAL, the task manager will filter it based on its unique_key. If the task corresponding to the unique_key has already been successfully processed, or the number of retries exceeds the preset maximum threshold, the task will not be passed to the downstream preprocessor and will be removed from the message queue and the WAL, thereby avoiding repeated processing.

[0091] Task timeout retry: The task manager will periodically scan the tasks in the WAL to check whether the processing time of the task exceeds the preset threshold. For tasks that exceed the threshold, the task manager will re-write them into the data stream to be re-consumed by the downstream preprocessor, thereby ensuring the final successful processing of the task.

[0092] This design ensures the data consistency and processing reliability of the system in the face of various failure conditions.

[0093] In an optional embodiment of the present application, the feature extraction on the video frame sequence obtains a video feature vector, including: writing the video frame sequence into a buffer queue; sequentially obtaining a current video frame from the buffer queue, and performing feature extraction on the current video frame to obtain a video feature vector.

[0094] In this embodiment, in order to solve the problem of GPU computing resource idling (i.e. GPU bubble) caused by the fluctuation of data output rate of the upstream preprocessor, a buffer queue is implemented inside each feature extractor Actor. This mechanism decouples data acquisition and GPU computation into two independent execution threads: Data acquisition thread: responsible for receiving tasks containing pre-processing results from the upstream, and putting video frame data (or its pointer on the GPU) into the buffer queue.

[0095] GPU computing threads: continuously take out the video frame data to be processed from the buffer queue and perform feature extraction computation.

[0096] Through this producer-consumer model, when the GPU completes a feature extraction, it can directly obtain the input data of the next task from the buffer queue for processing without waiting for the new data to be completely transmitted and ready from the upstream. This asynchronous processing method effectively hides the delay of data transmission, enabling the GPU to maintain high utilization and significantly improving the efficiency of feature extraction.

[0097] The entire feature extraction process is carried out in a streaming manner, which means that once the preprocessor generates enough video frame data, the feature extractor can start working immediately without waiting for the entire video preprocessing to be completed. This near-real-time processing method maximizes the reduction of data processing delay and enables the entire distributed feature storage and computing system to operate at high throughput and high resource utilization.

[0098] In an optional embodiment of the present application, the storing of the video feature vector into the distributed storage module comprises: serializing the video feature vector to obtain binary format feature data, and storing the binary format feature data into the distributed storage module.

[0099] In this embodiment, after completing feature extraction, in order to facilitate subsequent distributed storage and network transmission, the extracted feature data (usually a multi-dimensional floating-point tensor) is serialized into a compact binary format. The serialized binary data will be part of the task context and will be passed downstream to the distributed storage module in a streaming manner to store the binary format feature data into the distributed storage module.

[0100] In an optional embodiment of the present application, the obtaining of the video feature vector from the distributed storage module for model training or inference prediction in the model training stage or inference prediction stage comprises: receiving a model training task or inference prediction task for a video to be processed; querying the distributed storage module through a pre-set feature component whether there is a target feature vector matching the video to be processed; in the case that there is a matching target feature vector, then performing model training or inference prediction according to the target feature vector; in the case that there is no matching target feature vector, triggering an instant feature processing flow to extract a target feature vector corresponding to the video to be processed, and storing the target feature vector into the distributed storage module.

[0101] In this embodiment, by setting up a feature SDK, the application layer can directly obtain feature vectors from the distributed storage module for model training or inference computation. This feature SDK acts as a bridge between the distributed feature storage system and the application layer, providing a unified feature access interface. The core implementation of the feature SDK is as follows: Lazy computation interface (getOrCompute): First, it attempts to retrieve the computed features from distributed storage; if they do not exist, it triggers the computation process.

[0102] Batch Acquisition: Supports batch acquisition of multiple features, improving data acquisition efficiency.

[0103] Streaming access: Based on message queues, it provides streaming APIs and is suitable for scenarios that process large amounts of feature data.

[0104] Feature deserialization: Deserializes binary feature data into feature vectors for easier application.

[0105] The feature SDK allows for seamless integration of the training and inference frameworks into the system, skipping repetitive feature extraction steps and significantly reducing computational costs while improving efficiency.

[0106] By integrating this feature SDK, the inference and training frameworks can significantly save computational resources and development costs. Since feature extraction is a very time-consuming process, especially when dealing with large-scale video data, reusing pre-computed and stored features can significantly shorten model training and inference time and reduce the demand for computational resources.

[0107] The data feature storage and computation scheme in this embodiment can be implemented based on the Ray Data distributed computing framework, and the overall system architecture is as follows: Figure 4 As shown, Figure 4 The system comprises a data source module, a preprocessing module, a feature extraction module, a storage module, a feature SDK, and an application layer. The main steps of the data feature storage and processing flow through these modules include: 1. The data source module converts various inputs into a unified task format and schedules them through the task manager; 2. The preprocessing module receives the task, downloads the video and extracts frames, and then passes the video frame sequence to the feature extraction module. 3. The feature extraction module uses a dual-thread design and a buffer queue mechanism to efficiently perform GPU feature calculations; 4. The storage module persists the feature data to the distributed storage system; 5. The application layer can directly obtain the calculated features through the getOrCompute interface of the feature SDK, without having to calculate them again.

[0108] This streaming processing method ensures continuous data flow in each processing stage, avoids intermediate waiting, and improves the overall system processing efficiency.

[0109] The data feature storage and calculation scheme provided by the embodiment can uniformly and efficiently calculate and store video features in online scenarios (near real-time) and offline scenarios (offline batch), thereby significantly reducing the cost of model training and inference. By establishing a unified feature calculation and storage layer, the feature data of the video can be reused, the system efficiency is improved, and the computing resource consumption is reduced.

[0110] The embodiment overcomes the shortcomings of the background art through the following technical means: Decoupling of the calculation process: The process of processing videos by a multi-modal large language model is decoupled into preprocessing, feature extraction, and inference / training three independent links, realizing reasonable allocation and efficient use of computing resources.

[0111] Heterogeneous distributed computing: Based on the Ray framework, a heterogeneous and streaming distributed computing architecture is realized, and preprocessing and feature extraction are performed on CPU and GPU nodes respectively, and the number of computing nodes and batch size can be independently configured.

[0112] Streaming computing engine: A computing engine supporting stream batch integration is designed, which can simultaneously process real-time / near-line videos (message queue) and inventory videos (Hive table), realizing efficient use of computing resources.

[0113] Unified feature storage: A unified feature storage layer is constructed, and the extracted video features are stored in binary form in a distributed storage system (such as Iceberg, S3), realizing the persistence and reuse of feature data.

[0114] Feature SDK interface: A feature acquisition SDK is provided, supporting getOrCompute interface, so that the inference framework and the training framework can easily acquire the calculated feature data, avoiding repeated calculation.

[0115] Resource adaptive scheduling: Based on the task characteristics and resource usage, adaptive scheduling of computing resources is realized, maximizing resource utilization and computing efficiency.

[0116] Embodiment two Figure 5 The block diagram of the data feature storage and calculation device according to the second embodiment of the application is schematically shown, which can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the application. The program module referred to in the embodiments of the application refers to a series of computer program instruction segments capable of completing a specific function, and the functions of each program module in the embodiments will be specifically described below. For example,Figure 5 As shown, the data feature storage device 500 can include a data source module 501, a preprocessing module 502, and a feature storage module 503, wherein: The data source module 501 is configured to obtain video data from a plurality of different data source interfaces, and convert the video data into a preset task format to obtain a task stream containing meta information. The preprocessing module 502 is configured to perform frame extraction processing on the video data according to the task stream to obtain a video frame sequence. The feature storage module 503 is configured to perform feature extraction on the video frame sequence to obtain a video feature vector, and store the video feature vector and the meta information into a distributed storage module.

[0117] In an optional embodiment of the present application, the device further includes: The task management module is configured to write the meta information corresponding to the video data into a prewrite log module according to the task stream. Check the state of each preprocessing task in the task stream according to the unique key field in the meta information. In the case that the state of the preprocessing task is completed or exceeds the number of retries, an ACK confirmation message is generated.

[0118] In an optional embodiment of the present application, the task management module is further configured to: In the case that the state of the preprocessing task is to be processed, the preprocessing task is passed to the task stream, and after the preprocessing task is processed, the preprocessing task is deleted from the prewrite log module.

[0119] In an optional embodiment of the present application, the task management module is further configured to: Periodically scan the task stream in the prewrite log module to determine a timeout task, and re-pass the timeout task to the task stream to re-consume the timeout task.

[0120] In an optional embodiment of the present application, the feature storage module 503 includes: The feature extraction submodule is configured to write the video frame sequence into a buffer queue, and sequentially obtain a current video frame from the buffer queue and perform feature extraction on the current video frame to obtain a video feature vector.

[0121] In an optional embodiment of the present application, the feature storage module 503 includes: The feature storage submodule is configured to perform serialization processing on the video feature vector to obtain binary format feature data, and store the binary format feature data into the distributed storage module.

[0122] In an optional embodiment of the present application, the device further comprises: The feature acquisition module is configured to receive a model training task or an inference prediction task for a to-be-processed video; query the distributed storage module through a preset feature component to determine whether there is a target feature vector matching the to-be-processed video; if there is a matching target feature vector, then perform model training or inference prediction according to the target feature vector; if there is no matching target feature vector, then trigger an instant feature processing procedure to extract a target feature vector corresponding to the to-be-processed video, and store the target feature vector in the distributed storage module.

[0123] Embodiment Three Figure 6 A hardware architecture schematic diagram of a computer device 10000 suitable for implementing the data feature storage and calculation method according to Embodiment Three of the present application is schematically shown. In some embodiments, the computer device 10000 can be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workstation, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 can be a rack-mounted server, a blade server, a tower server, or a cabinet server (including a standalone server, or a server cluster composed of multiple servers), etc. As shown in the figure, the computer device 10000 includes but is not limited to a memory 10010, a processor 10020, and a network interface 10030 which are communicatively linked through a system bus. Among them: Figure 6 ​The memory 10010 includes at least one type of computer-readable storage media, such as a flash memory, a hard disk, a multimedia card (e.g., SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 10010 can be an internal memory module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external memory device of the computer device 10000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Of course, the memory 10010 can include both an internal memory module and an external memory device of the computer device 10000. In this embodiment, the memory 10010 is generally used to store an operating system and various application programs installed in the computer device 10000, such as program codes of the data feature storage method, and the like. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0124] The processor 10020 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips in some embodiments. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication of the computer device 10000, and the like. In this embodiment, the processor 10020 is used to run program codes or process data stored in the memory 10010.

[0125] The network interface 10030 can include a wireless network interface or a wired network interface, and is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 with an external terminal through a network, establish a data transmission channel and a communication link between the computer device 10000 and the external terminal, and the like. The network can be an Intranet, the Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, and the like wireless or wired network.

[0126] It should be noted that, Figure 6 Only the computer device with the components 10010-10030 is shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0127] In this embodiment, the data feature storage method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as the processor 10020) to complete the data feature storage method of the embodiments of the present application.

[0128] Embodiment Four The embodiments of the present application also provide a computer readable storage medium, and the computer readable storage medium has a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the data feature storage method in the embodiments.

[0129] In this embodiment, the computer readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a programmable read only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer readable storage medium can be an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the computer readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer readable storage medium is usually used to store an operating system and various application software installed on the computer device, for example, program codes of the data feature calculation method in the embodiments, etc. In addition, the computer readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0130] Embodiment five The embodiments of the present application further provide a computer program product, comprising a computer program which, when executed by a processor, implements the method in the above embodiments.

[0131] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the present application can be implemented by using a general computer device, which can be concentrated on a single computer device or distributed on a network composed of multiple computer devices, and optionally, each module or each step can be implemented by using program codes executable by a computer device, so that each module or each step can be stored in a storage device and executed by a computer device, and in some cases, the steps shown or described can be executed in an order different from that shown here, or each module or each step can be manufactured into an individual integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Therefore, the embodiments of the present application are not limited to any particular combination of hardware and software.

[0132] It should be noted that the above is only the preferred embodiment of the present application, and does not limit the patent protection scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data feature storage and computation method, characterized in that, The method includes: Video data is obtained from multiple different data source interfaces, and the video data is converted into a preset task format to obtain a task stream containing metadata. The video data is processed by frame extraction according to the task flow to obtain a video frame sequence; The video frame sequence is subjected to feature extraction to obtain a video feature vector, and the video feature vector and metadata are stored in a distributed storage module.

2. The data feature storage and computation method according to claim 1, characterized in that, After the step of converting the video data into a preset task format to obtain a task stream containing metadata, the method further includes: The metadata corresponding to the video data is written into the write-ahead log module according to the task flow; The status of each preprocessing task in the task flow is checked based on the unique key field in the metadata. If the status of the preprocessing task is completed or the number of retries has been exceeded, an ACK confirmation message is generated.

3. The data feature storage and computation method according to claim 2, characterized in that, The method further includes: When the status of the preprocessing task is "needs processing", the preprocessing task is passed to the task flow, and after the preprocessing task is completed, the preprocessing task is deleted from the write-ahead log module.

4. The data feature storage and computation method according to claim 2, characterized in that, The method further includes: The task flow in the write-ahead log module is periodically scanned to identify timed-out tasks, and the timed-out tasks are retransmitted into the task flow for re-consumption.

5. The data feature storage and computation method according to claim 1, characterized in that, The step of extracting video feature vectors from the video frame sequence includes: Write the video frame sequence into a buffer queue; The current video frame is sequentially retrieved from the buffer queue, and the current video frame is used to extract features to obtain a video feature vector.

6. The data feature storage and computation method according to claim 1, characterized in that, The step of storing the video feature vector into the distributed storage module includes: The video feature vector is serialized to obtain binary format feature data, and the binary format feature data is stored in the distributed storage module.

7. The data feature storage and computation method according to claim 1, characterized in that, The method further includes: Receive model training tasks or inference prediction tasks for the video to be processed; The system queries the distributed storage module to determine whether a target feature vector matching the video to be processed exists in the pre-set feature components. If a matching target feature vector exists, then model training or inference prediction is performed based on the target feature vector; If no matching target feature vector exists, an instant feature processing flow is triggered to extract the target feature vector corresponding to the video to be processed, and the target feature vector is stored in the distributed storage module.

8. A data feature storage and computing device, characterized in that, The device includes: The data source module is used to obtain video data from various different data source interfaces and convert the video data into a preset task format to obtain a task stream containing metadata. The preprocessing module is used to perform frame extraction processing on the video data according to the task flow to obtain a video frame sequence; The feature storage module is used to extract features from the video frame sequence to obtain video feature vectors, and store the video feature vectors and metadata in the distributed storage module.

9. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data feature storage and calculation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the data feature storage and calculation method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the data feature storage and calculation method as described in claims 1 to 7.

Images