Data processing method and device, equipment, storage medium and computer program product

Abstract

The application provides a data processing method, device, equipment, storage medium and computer program product. The method comprises the following steps: in response to a data service request of a request object, determining a data service type; determining a matching query module matched with the data service type; if the matching query module is a first type of query module, obtaining matching medium feature data from a corresponding storage module and obtaining target data; and returning the target data to the request object. A data processing system comprises a plurality of query modules for executing a plurality of types of data services. The system further comprises a data main storage module for storing medium content data and medium feature data, and generating feature data change flow based on a change event of the medium feature data. The medium feature data stored in the storage module is determined based on the feature data change flow synchronized from the data main storage module by the query module. The application can support data query of multiple types of data services, and can effectively improve data query efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods, data processing apparatus, computer equipment, computer-readable storage media, and computer program products. Background Technology

[0002] With the continuous development of the internet, the data from various businesses is growing rapidly, necessitating the storage of this business data and meeting the data writing and querying requirements under different business scenarios. The demand for storing large amounts of business data drives the performance requirements of the content library to continuously increase. High-concurrency data queries also put enormous pressure on the database system, causing instability, high latency, and low data query efficiency. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, storage medium, and computer program product that can support data querying for multiple types of data services and effectively improve the efficiency of data querying.

[0004] Firstly, this application provides a data processing method, including:

[0005] In response to a data service request from a requesting object, determine the data service type of the data service request;

[0006] Determine the matching query module that matches the data service type from among the configured multiple query modules;

[0007] If the matching query module is a first type of query module, then the matching query module is called to obtain the matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and obtain the target data requested by the data service request based on the matching media feature data.

[0008] Return the target data to the requesting object;

[0009] Each query module is used to execute one type of data service, and multiple query modules correspond to multiple types of data services. Multiple query modules are included in the data processing system, which also includes a data main storage module. The data main storage module is used to store media content data and media feature data, as well as to generate feature data change streams based on change events of media feature data. The media feature data stored in the storage module is determined based on the feature data change streams synchronized from the data main storage module by the query modules.

[0010] Secondly, this application provides a data processing apparatus, comprising:

[0011] The processing unit is used to respond to the data service request of the requesting object and determine the data service type of the data service request;

[0012] The processing unit is used to determine the matching query module that matches the data service type from the configured multiple query modules; if the matching query module is a query module of the first type, the matching query module is called to obtain the matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and the target data requested by the data service request is obtained based on the matching media feature data.

[0013] The transceiver unit is used to return target data to the requesting object;

[0014] Each query module is used to execute one type of data service, and multiple query modules correspond to multiple types of data services. Multiple query modules are included in the data processing system, which also includes a data main storage module. The data main storage module is used to store media content data and media feature data, as well as to generate feature data change streams based on change events of media feature data. The media feature data stored in the storage module is determined based on the feature data change streams synchronized from the data main storage module by the query modules.

[0015] Thirdly, this application provides a data processing device, including a processor, a communication interface, and a memory, which are interconnected. The memory stores executable program code, and the processor is used to call the executable program code to implement the data processing method provided in this application.

[0016] Fifthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to implement the data processing method provided in this application.

[0017] Sixthly, this application provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to implement the data processing method provided in this application.

[0018] The data processing method provided in this application is implemented based on a data processing system that includes a main data storage module and multiple query modules. The main data storage module is used to store media data and media feature data for various media, and to generate feature data change streams based on change events of the media feature data. Each query module is used to execute a type of data service, and the media feature data stored in the storage module corresponding to each query module is determined based on the feature data change stream synchronized by the query module from the main data storage module. Therefore, by constructing a main data storage module and configuring multiple query modules, this application adopts different storage schemes for the business data required by various data services, forming a Command Query Responsibility Segregation (CQRS) architecture. This enables the data processing system to respond to business requests from various data services and maintains stable service performance even under high concurrency. Furthermore, the query modules synchronize data through the data change stream of the main data storage module, which reduces data synchronization latency and ensures data consistency between the main data storage module and the query modules.

[0019] In its specific implementation, the data processing method provided in this application determines the data service type of the data service request of the requesting object, and selects a matching query module from multiple query modules configured in the data processing system that matches the data service type. If the matching query module is a query module of the first type, it can be called to retrieve the matching medium feature data matching the service request from the corresponding storage module, and obtain the target data requested by the data service request based on the matching medium feature data, and return the target data to the requesting object. Therefore, this application allocates different query modules for different types of data service requests, which not only supports data querying for multiple types of data services, but also enables synchronous data querying for multiple types of data services through different query modules, thereby effectively improving the efficiency of data querying. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a data processing system provided in an embodiment of this application;

[0023] Figure 3a A flowchart illustrating a data processing method provided in an embodiment of this application;

[0024] Figure 3b A schematic diagram illustrating a data service request result provided in an embodiment of this application;

[0025] Figure 4a A flowchart illustrating another data processing method provided in an embodiment of this application;

[0026] Figure 4b This application provides a schematic diagram of the data main memory module writing process for an embodiment of the present application.

[0027] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] To facilitate understanding, the terms used in this application will be explained first.

[0031] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, be used on demand, and is flexible and convenient. Cloud computing technology will become a crucial support. In this application, the backend services of the data processing system require substantial computing and storage resources, such as those required for video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0032] A database can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or Extensible Markup Language (XML); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as Structured Query Language (SQL); or according to performance priorities, such as maximum scale or highest operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​simultaneously.

[0033] High concurrency is a crucial factor to consider in the design of distributed system architectures for the internet. It typically refers to designing a system to handle a large number of requests simultaneously. Common metrics related to high concurrency include response time, throughput, queries per second (QPS), and concurrent users. Response time refers to the time it takes for the system to respond to a request; throughput refers to the number of requests processed per unit of time; QPS refers to the number of requests responded to per second; and concurrent users refer to the number of users simultaneously using the system's functions. For example, in a news application (APP), the number of users online at the same time represents, to some extent, the system's concurrent user count.

[0034] Content repositories are the data support for building online media and a key component and technology in constructing news and other online media. With the popularization of online media, content repositories, as the data support, have become a key focus of technological research. Typically, content repositories use a single physical storage medium to support different types or media of data, thereby building and realizing unified content service capabilities for multimedia applications. The main focus is on how to build and implement large-capacity, high-speed, multi-media integrated storage and retrieval capabilities. For example, a content repository built on an Elastic Search (ES) server can be used to query and retrieve content within the repository; another example is a content repository primarily using traditional relational databases, focusing on relational database design.

[0035] Existing content library systems typically cater to relatively simple business logic scenarios, employing a single storage method to cover the data writing and querying needs of online media applications. However, in high-concurrency scenarios, using a single database may not guarantee stable service quality. For example, news apps may have tens of millions of daily active users. In such cases, the performance requirements for the content library service are much higher. Besides meeting high-concurrency data read / write demands, there's also the need for efficient data retrieval, as well as extensive data analysis and mining, often requiring scanning the entire content library. These various data services place extremely high demands on the content library's data query efficiency and latency, which a single storage system cannot meet. For instance, to achieve retrieval of multiple fields, multiple indexes are needed, making the data table structure overly complex and hindering the achievement of high read / write speeds. Furthermore, frequent, large-scale database scans are required. Therefore, a single storage system cannot simultaneously meet multiple complex objectives such as multi-source access, complex retrieval, historical data backtracking, efficient access, and online / offline analysis.

[0036] Different business scenarios have different data query needs. For example, news content management backend systems often need to search for related content based on specified conditions, which requires the content repository to have content retrieval capabilities. When managing a business, it is usually necessary to measure the content supply and the time it takes for data content to flow, so as to measure the current status of the business from a global perspective. This requires the content repository to support data analysis capabilities. Recommendation indexes require the consumption of distributable data, which requires filtering of distributable content before data is distributed, and also requires support for full distribution to support full construction. This requires the content repository to have streaming distribution and full construction capabilities.

[0037] Different business scenarios have vastly different data query requirements. Implementing multiple data query requirements in one scenario would result in extremely high system design complexity, and even if it were implemented, it would be difficult to guarantee high stability and performance.

[0038] To address the aforementioned problems, this application provides a data processing system for executing data processing methods and providing various data services, such as... Figure 1 As shown, the data processing system includes a main data storage module 110, a query module 120, and a data production module 130. The main data storage module 110 and multiple query modules 120 form a content library. The data processing system establishes a network connection with a data request terminal 140 and performs data interaction. This data processing system may include multiple query modules 120, and the data request terminal 140 may include one or more request objects. It should be noted that... Figure 1 The number and configuration of devices in the data processing system shown are for illustrative purposes only and do not constitute a limitation on the embodiments of this application.

[0039] The data production module 130 can process and calculate features of various content in the data processing system, such as images, videos, and articles, to generate media content and media feature data, and then write the obtained media content and media feature data into the main data memory module 110. The data production module 130 may include a feature scheduling pipeline to provide orchestration and scheduling of the content feature processing flow, thereby enabling the media content and media feature data to be written into the main data memory module in an orderly manner. The data production module 130 can also connect to external data sources, enabling multi-source data access.

[0040] The main data storage module 110 can receive media content data and media feature data generated by the data production module 130, as well as feature data change streams generated based on change events of the media feature data. The main data storage module 110 can send the generated feature data change streams to various query modules. The configuration center in the main data storage module 110 can configure event identifiers (Identity Documents, IDs) for the generated data change streams, sorting the data change streams according to the value of the event ID to ensure the orderliness of subsequent processing of the data change streams. The main data storage module 110 may include a pipeline generator for generating feature data change streams based on change events of the media feature data.

[0041] The query module 120 can be a combination of various types of query modules. Each query module is used to execute a type of data service, and multiple query modules correspond to multiple types of data services. The query module can obtain media characteristic data through the characteristic data change stream generated by the main data storage module 110, and store the obtained media characteristic data in the storage module corresponding to the query module.

[0042] The data request terminal 140 can send various types of business requests to the data processing system. These requests may carry the data business type, or the data business type may be obtained by the data processing system through parsing the business request. The data request terminal 140 can be a smart terminal configured with applications such as news apps, capable of initiating business requests to the data processing system. For example, if a user has a news app installed on their smartphone, when the user launches the app, the app can initiate a business request to the data processing system, such as a homepage news recommendation. After determining the request type, the data processing system can call a query module matching that request type to retrieve the target data from the corresponding storage module and send it to the news app, creating a recommendation page on the app's homepage for the user to view.

[0043] The data processing system provided in this application embodiment can be implemented based on one or more of cloud technology, artificial intelligence technology, and blockchain technology. For example, it can involve one or more of cloud technologies, such as cloud storage and cloud database. For instance, at least a portion of the data involved in performing the data processing method (e.g., media content data and feature data of various media) can be stored in a cloud database. Alternatively, at least a portion of the data involved in performing the data processing method can be stored on a blockchain in the form of blocks; furthermore, the computer device performing the data processing method can be a node device in a blockchain network.

[0044] It can be seen that, as Figure 1 The data processing system shown implements a Command Query Responsibility Segregation (CQRS) architecture by constructing multiple query modules to execute various data operations, while the main data storage module is primarily used for data storage. CQRS is an architectural pattern that separates commands that change the model's state from queries that modify the model's state. The command side is responsible for data updates, while data queries are primarily handled by a separate query side. Updates and queries are implemented by different modules, thus simplifying the complexity of the system architecture.

[0045] Based on such Figure 1 The diagram illustrates the structure of a data processing system, and is a scenario diagram of a data processing system provided in this application embodiment. Please refer to... Figure 2 The data processing system may include, but is not limited to, one or more terminals 210 and one or more servers 220. Terminals 210 and servers 220 establish a communication connection and exchange data via a wired or wireless network. It should be noted that... Figure 2 The number and form of the devices shown are for illustrative purposes only and do not constitute a limitation on the embodiments of this application.

[0046] In this embodiment, terminal 210 may include, but is not limited to, smart devices such as smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, in-vehicle terminals, smart home appliances, smart voice interaction devices, and aircraft.

[0047] In this application, terminal 210 can be used as a request object to send business requests to server 220 and receive target data sent from server 220. For example, terminal 210 can be a computer used to manage a news app. Administrators can send business requests to server 220 through this terminal, such as calculating the readership of news articles written by contracted authors of the news app in the first quarter. Server 220 can then match an appropriate query module based on the business type to process the data and return the matched target data to the terminal.

[0048] In this embodiment, the server 220 in the data processing system can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0049] In this application, server 220 stores media content data and media feature data of the media, and generates feature data change streams based on change events of the media feature data. Server 220 can also be used to receive and respond to service requests sent by a requesting object (which may be terminal 210), determine the type of data service requested by the requesting object based on the service request, call a query module that matches the data service type, retrieve matching media feature data that matches the data service request from the storage module corresponding to the matching query module, obtain target data based on the matching media feature data, and return the obtained target data to the requesting object.

[0050] Server 220 may be configured with, for example: Figure 1 The data processing system shown includes a main data storage module and multiple query modules. Alternatively, server 220 can also be configured as follows: Figure 1The data processing system shown has multiple query modules that can be used to obtain media characteristic data based on data change streams generated by other servers in the distributed system that are configured with data main storage modules, and store the data in the storage module corresponding to the query module. When a query module that matches the data service type is called, the matching query module configured in server 220 can obtain matching media characteristic data that matches the data service request from the corresponding storage module, obtain the target data based on the matching media characteristic data, and return the obtained target data to the request object.

[0051] The above provides a brief overview of the data processing method provided in the embodiments of this application. The specific implementation of this data processing method will be described in detail below.

[0052] The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0053] It is understood that in the specific implementation of this application, data related to data service requests, media content, media feature data, etc. are involved. When the above embodiments of this application are applied to specific products or technologies, the relevant data must obtain the permission or consent of the relevant parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0054] Based on the aforementioned data processing system, this application provides a data processing method, applicable to scenarios such as news content libraries and converged media content libraries. The execution entity of this method is a data processing device. Figure 3a As shown, taking the application scenario of a news content library as an example, this data processing method includes, but is not limited to, the following steps:

[0055] S301: In response to a data service request from a requesting object, determine the data service type of the data service request.

[0056] In this application, the requesting object can be a terminal, such as a smartphone or computer configured with data service-related applications. In this case, the data service type requested could be a news keyword search service for a news application. The requesting object can also be a server, such as the backend server of a news app. In this case, the data service type requested could be the number of active users in the app during a certain period of time.

[0057] Optionally, the data service request may carry the data service type, and the data processing device can directly obtain the corresponding data service type from the data service request.

[0058] Optionally, data service types can include various types such as data analysis, data retrieval, data recommendation, content browsing, and historical data query. Specifically, data analysis can analyze which types of news articles are most popular with readers based on data from a recent period (such as readership, likes, etc.); data retrieval can search for data within a category (such as current affairs, weather, etc.) to identify and manage any violations; data recommendation can build a series of recommended content based on users' historical browsing records; content browsing can provide real-time content viewing under high concurrency to reduce business response latency; and historical data query can provide users with the service to view data from a specific point in time, such as viewing news articles published on a particular day in 2008 and related comments from that time in 2022.

[0059] S302: Determine the matching query module that matches the data service type from among the configured multiple query modules.

[0060] In this application, the data processing system is configured with multiple query modules and a main data storage module. The main data storage module stores media content data and media feature data, and generates feature data change streams based on change events of media feature data. The query modules can synchronize data from the data change streams generated by the main data storage module. Different types of query modules can select appropriate storage media (such as Elasticsearch) to store the synchronized feature data in the corresponding storage module according to the functions and requirements of data queries in different scenarios. The query module can be a materialized read view built based on the data in the main data storage module. A materialized read view is a database object including a query result; it is a local copy of remote data, or it can be used to generate a summary table based on the summation of data tables, and can be used to store data based on remote tables. Each query module is used to execute one type of data service, and multiple query modules correspond to multiple types of data services.

[0061] In this application, "media" refers to data content in various forms of presentation, such as images, articles, and videos involved in news. "Media content data" refers to the news content included in the aforementioned images, articles, and videos. "Media feature data" refers to the data characteristics of the media content data, such as news release time, author, number of reads, number of reposts, and type of news category.

[0062] The data change stream can be generated by the main data storage module using an event log approach, based on the media feature data before the change and the acquired media feature data after the change. Each time the media feature data stored in the main data storage module undergoes a change, such as an author modifying text content in a published news article, a change event log is recorded. The change event log includes the time of the change, the object of the data change, the changed feature data, and the value of that feature data after the change. The change event log is stored append-only, preventing overwriting. Therefore, all data change records can be retrieved from the recorded change event log, enabling the data processing system to have historical backtracking capabilities. Based on all currently recorded change event logs, data recorded at any point in time can be retrieved. Synchronizing data changes through the data change stream reduces the synchronization latency to the millisecond level, which is imperceptible to the actual business system. Therefore, all data changes can be considered strongly consistent between the main data storage module and various query modules.

[0063] Optionally, to accelerate the acquisition of the latest feature data for the article, we also store snapshots, which are overwritten each time the article event log is updated. The snapshot stores the latest feature data for the article.

[0064] In one implementation, the media feature data may include static feature data and dynamic feature data. Static feature data refers to feature data with low change frequency, such as the text content of news articles, the accompanying pictures in news content, and videos. Dynamic feature data refers to feature data that changes frequently, such as the number of views of articles, the number of plays of videos, and the number of comments received.

[0065] Since frequently changing feature data and the resulting change event logs can put significant storage pressure on the main memory module, the main memory module can optionally include a main memory database and a dynamic feature unit. The main memory database stores static feature data and change event logs generated based on changes to static feature data. The dynamic feature unit is a storage module independent of the main memory, storing dynamic feature data and change event logs generated based on dynamic feature data. This achieves isolated storage of static and dynamic feature data, reducing storage pressure on the main memory module and preventing main memory capacity expansion.

[0066] In one implementation, the dynamic feature unit can be a storage unit contained within the main memory database. Feature storage is unified in the main memory database, and the main memory database is built with adaptive compression capabilities for change event logs: for frequently changing feature data, the change log is recorded for each change, and the change event log is compressed through a separate background process; for infrequently changing feature data, no compression is required. Based on this, the storage performance of the main memory database can be expanded, achieving unified storage of feature data.

[0067] The main storage database can adopt a key-value (KV) storage model with a schema-less design, supporting data writing in various forms and formats. This model has relatively simple logic, provides high-performance data writing, and is stable and reliable. It solves many data fragmentation storage problems and reduces the customized development and maintenance costs caused by heterogeneous business operations.

[0068] In one implementation, when storing changes to dynamic feature data, the dynamic feature unit can sample these changes according to a fixed time window, such as sampling every hour. The sampled, changed dynamic feature data is then stored in the main memory database, allowing the main memory module to generate a data change log based on the change event log. This avoids frequently recording feature data changes and reduces the storage pressure on the main memory database.

[0069] In one implementation, the main data storage module can generate three types of data change streams based on the media feature data before the change and the acquired media feature data after the change: incremental change stream, full-text stream, and change comparison stream. The incremental change stream may include the changed media identifier (Identity Document, ID), the changed feature data ID, and the value of the changed feature data; the full-text stream may include the changed media identifier, the changed media content data, and the value of the media feature data; and the change comparison stream may include the changed media identifier, the changed media feature data, the value of the media feature data before the change, and the value of the media feature data after the change. Each generated data change stream has an event ID, which is unique and increments over time, using a snowball algorithm. By comparing the event IDs of each data change stream, the chronological order of the data change streams can be determined based on the magnitude of the event ID values, ensuring the orderly processing of subsequent data change streams.

[0070] In one implementation, data consistency between the main storage module and various query modules can be guaranteed by synchronizing the data change stream. Optionally, Kafka's message queue partitioning mechanism can be used to ensure the ordered delivery of the data change stream. After generating the data change stream, the main storage module uses a consistent hashing algorithm to calculate a partition ID based on the media identifier. Data change streams for the same media (such as a news article) are placed in the same partition. Based on the message queue's operating mechanism, a partition will only be consumed by one query module. Each query module can run several concurrent coroutines to improve processing speed. After the query module consumes the data change stream, it can perform fixed mappings across multiple concurrent coroutines based on the media identifier in the data change stream, ensuring that a data change stream for one media is only processed by one coroutine within a single query module. This ensures that data change streams for the same media are processed serially, strictly guaranteeing the order of data change stream processing, ensuring the correctness of data updates in the query modules, and achieving data consistency.

[0071] In this application, the data processing system can be configured with multiple query modules, each designed to satisfy data service requests in a specific application scenario. Each query module's corresponding storage module stores data adapted to that data service type, and the system can retrieve the target data requested by the requesting object from the storage module. Multiple query modules can be configured to execute various types of data services, thereby distributing the data services provided by the data processing system to multiple modules for processing, which improves the system's service capabilities and efficiency for various data services.

[0072] S303: If the matching query module is a query module of type 1, then the matching query module is called to obtain the matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and obtain the target data requested by the data service request based on the matching media feature data.

[0073] Each query module's corresponding storage module stores multiple media feature data groups. Each feature data group includes static and dynamic feature data for a media. For example, static feature data such as the release time, publisher, and video duration of a news video, and dynamic feature data such as video views and likes. Furthermore, the first type of query module refers to a query module that needs to process feature data to provide services, and may include a data analysis module, a data retrieval module, and a data recommendation module.

[0074] Optionally, the data service type can be a data analysis service type. For example, when managing a certain service of a news app, the supply of news content can be measured to assess the current status of the service from a global perspective. The data service request can carry the feature identifier to be analyzed (such as the number of views of a news video) and the media identifier to be analyzed (such as the playback link of the news video). In this case, the matching query module matching this data service type can be a data analysis module (or analysis view). The static feature data stored in the storage module corresponding to the data analysis module includes the media identifier. In response to the data service request, the data analysis module can be invoked to determine the first media feature data group containing the media identifier that matches the media identifier to be analyzed from multiple media feature data groups stored in the storage module corresponding to the data analysis module; the matching media feature data that matches the feature identifier to be analyzed can be obtained from the first media feature data group; data analysis can be performed on the matching media feature data to obtain the analysis result, and the analysis result can be determined as the target data requested by the data service request. It should be noted that the data stored in the data analysis module's storage module is time-sensitive and can be configured according to specific application scenarios. For example, it can be configured to only store data within the last six months, and data older than six months will be deleted.

[0075] The data analysis module's corresponding storage module can be an offline data storage module such as Hive or Iceberg, built based on big data streaming computing frameworks (Flink). This can meet the needs of both real-time and offline analysis, while simultaneously outputting content tables to the data management team of the data processing system to build a news content data warehouse. Iceberg is an open table format for massive data analysis scenarios; Hive is a data warehouse tool based on Hadoop (a distributed system infrastructure), a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop, which can be used for data extraction, data transformation, and data loading. The advantages of using Hive technology are: relatively low construction cost; support for index creation, which optimizes data queries; support for different storage types, such as plain text files and files in HBase; and storing data in a relational database, which can greatly reduce the time spent performing semantic checks during queries.

[0076] Optionally, the data service type can be a data retrieval service type. For example, a news content management backend system often needs to search for related content based on specified conditions. The data service request can carry search keywords, such as "2022 National Basketball Championship". In this case, the matching query module matching this data service type can be a data retrieval module (or a retrieval view). The static feature data stored in the storage module corresponding to the data retrieval module includes media identifiers and media tags. The media tags can refer to the category of the media, such as sports news, entertainment news, etc. In response to the data service request, the data retrieval module is invoked. From the multiple media feature data groups stored in the storage module corresponding to the data retrieval module, a second media feature data group containing media tags that match the search keywords is determined. The target media identifier contained in the second media feature data group is obtained. This target media identifier is the matching media feature data that matches the data service request. The media content data corresponding to the target media identifier is obtained from the main data storage module, and the obtained media content data is determined as the target data requested by the data service request. It should be noted that the data stored in the data retrieval module is time-sensitive and can be configured according to specific application scenarios. For example, it can be configured to only store data within the last six months, and data older than six months will be deleted.

[0077] The storage module corresponding to the data retrieval module can be a storage module built on the Elasticsearch (ES) database. ES is also a distributed document database where every field is indexable and searchable, capable of horizontal scaling to hundreds of servers and processing petabytes (1 PB = 1024 TB) of data. It can store, search, and analyze massive amounts of data in a very short time. It is typically used as the core engine in scenarios with complex search requirements. Applied to this application, it can achieve efficient data retrieval or data search.

[0078] Optionally, the data service type can be a data recommendation service type, and the matching query module can be a data recommendation module (or distribution view). The static feature data stored in the storage module corresponding to the data recommendation module may include media identifiers, media tags, and publication time. In response to this data service request, the data recommendation module can be invoked. Based on the media tags (sports, entertainment, etc.) in multiple media feature data groups stored in the storage module, the module determines the media feature data groups that meet the filtering conditions and performs filtering processing on the media feature data groups that meet the filtering conditions. The data recommendation module can also determine the recommended media identifier from the unfiltered media feature data groups stored in the storage module based on the media tags and publication time. The recommended media identifier is the matching media feature data (sports events, sports stars; entertainment award ceremonies, entertainment stars, etc.) that matches the data service request. The module then retrieves the media content data corresponding to the recommended media identifier from the main data storage module and determines the retrieved media content data as the target data requested by the data service request. Here, the filtering conditions may refer to the search keyword A involving illegal or unhealthy content. Filtering can involve deleting media feature data groups that meet the filtering criteria, or adding a filtering identifier to media feature data groups that meet the filtering criteria, so that when determining the target data, media feature data groups that meet the filtering criteria are excluded from the target data.

[0079] In one implementation, the query module matching the data service type can also be a second type of query module. This second type of query module can be one that does not require processing based on feature data to provide services. Specifically, the second type of query module may include a content browsing module and a historical data query module.

[0080] Optionally, the data service type can be a content retrieval service type. If the data service request carries a media identifier to be retrieved, then the query module matching this data service request is the content retrieval module (or cached view). The content retrieval module can synchronize media content data from the main data storage module and store the synchronized media content data in the corresponding cache module. In response to the data service request, the content retrieval module is invoked to query the media content data corresponding to the media identifier to be retrieved from the cache module; the media content data corresponding to the media identifier to be retrieved retrieved from the cache module is determined to be the target data requested by the data service request.

[0081] Since the data cached by the caching module is time-sensitive, the time window for caching data can be set according to different application scenarios. For example, it can be set to cache only data within the last three months and clear data older than three months. In such cases, if the media content data corresponding to the media identifier to be queried cannot be found in the caching module, the media content data corresponding to the media identifier to be queried can be obtained from the main memory database.

[0082] The caching module differs from the storage module mentioned earlier. It can be a storage device built on Redis, which can support high-concurrency access to online content, with an average query latency of less than 10 milliseconds and a single node capable of supporting 100,000 query QPS.

[0083] Optionally, the data service type can be a historical data query service type. The data service request can carry the media identifier to be queried, the feature identifier to be queried, and the time to be queried. The query module matching this data service type is the historical data query module. In response to the data service request, the historical data query module is invoked to query the change event log of the media feature data corresponding to the media identifier to be queried from the main data storage module. Based on the queried change event log of the media feature data, the historical media feature data matching the feature identifier and the time to be queried is determined. The historical media feature data is determined as the target data requested by the data service request. Since the change event log stored in the main data storage module is stored in an append-only manner, there is no overwriting between the change event logs, so historical data at any recorded time point can be obtained.

[0084] The historical data query module enables online issue investigation and provides data rollback capabilities. For example, it can be used to quickly remove dirty data written by a specified business party within a specified time window, achieving rapid data rollback and quick restoration of business services. Dirty data refers to data that is outside the given scope, has no meaning for actual business operations, is in an illegal format, or contains non-standard coding and ambiguous business logic.

[0085] The query module is flexibly expandable. In one implementation, new first-type or second-type query modules can be constructed based on other business scenarios; this application does not limit this. For example, a time-series view can be constructed, which is a first-type query module. Its corresponding storage module can be used to collect all media content data and media feature data in chronological order, and data analysis can be performed based on the data stored in the storage module. For example, based on the historical changes in the readership of a news article, trend analysis can be performed to predict the future readership trend of the news article.

[0086] S304: Return the target data to the requested object.

[0087] In this application, since the requesting objects have different query methods and format requirements for the target data, the matching query module can adapt and convert the data format according to the data business type of the requesting object to meet the data format requirements of different requesting objects.

[0088] In one implementation, the matching query module can return target data to the requesting object based on a message queue, thereby ensuring the orderliness of the returned target data.

[0089] For example, if the requested object is a news webpage on a computer device, and the requested data service type is data retrieval, then after the data processing system responds to the request and determines the target data, it can return the target data to the news webpage and display it according to the matched data format. Please refer to [link / reference]. Figure 3b This is a schematic diagram illustrating a data service request result scenario provided in an embodiment of this application. A user can log in to a news webpage and, as shown in the diagram... Figure 3b In the search bar of the news webpage shown in Figure 310, users can enter search keywords, such as "sports." Once the news webpage receives the user's input, it generates a corresponding data service request and sends it to the data processing system. Ultimately, it obtains target data related to that search keyword, such as the sports-related search results list shown in the figure, which may include... Figure 3b The list of news articles shown in 320 and such Figure 3b The image in section 320 shows related videos. Users can click on the title of the search result they are interested in, such as... Figure 3b The news headline shown in 340 provides the full content of the news article; users can also click on other relevant links. Figure 3b Click the play button on the news video shown in image 350 to watch the video. Additionally, users can click on other buttons such as... Figure 3b Click "View More" in 360 to get more news content related to your search term.

[0090] As can be seen, this application allocates different query modules for data querying based on different types of data service requests. This not only supports data querying for multiple types of data services, but also enables synchronous data querying for multiple types of data services through different query modules, thereby effectively improving the efficiency of data querying.

[0091] Please see Figure 4a , Figure 4a This is a flowchart illustrating another data processing method provided in an embodiment of this application, wherein the execution subject of this method is a data processing device. Figure 4a As shown, taking the application scenario of a news content library as an example, this data processing method includes, but is not limited to, the following steps:

[0092] S401: Access the media data content and media feature data of various media generated by the data production module.

[0093] The data production module, which can be included in the data processing system, can process and calculate features of various types of content, such as images, videos, and articles, to generate media content and media feature data, and then write the obtained media content and media feature data into the main data memory module. The data production module may include a feature scheduling pipeline to provide orchestration and scheduling of the content feature processing flow, thereby ensuring that media content and media feature data are written to the main data memory module in an orderly manner.

[0094] For example, Figure 4b This is a schematic diagram illustrating the data writing process of a data main storage module provided in an embodiment of this application. The data production module may include multiple data writers (such as authors or news agencies publishing news in a news app), and the data main storage module can also control data access, for example, based on settings such as... Figure 4b The security barrier logic shown, and the security review rules (such as...) Figure 4b As shown in Figure 470), the system determines whether the data writer is legitimate and whether the written characteristics are registered legitimate characteristics. The main data storage module provides both synchronous and asynchronous data access methods for the data production module. Asynchronous access is via a Kafka message queue, suitable for large-scale data updates; synchronous access is via a request interface, synchronously returning the write result. The main data storage module can subscribe to data messages from the data production module and generate events through message parsing, from existing snapshot tables (…). Figure 4b Query the corresponding snapshot (as shown in 410) in the table. Figure 4b As shown in Figure 420, the system compares the data with existing data. If there is no difference, the write operation is considered invalid and the write operation ends. If there is a difference, a snapshot is generated to record the latest data. When writing data to the main memory module, the data writer's identifier (such as the news agency's identity) must be included. The data processing system configures the priority of each data writer. Data written by a lower-priority data writer cannot overwrite data written by a higher-priority data writer. If manual intervention is required, data writing can be performed through the manual interface shown in the figure. The writing priority is higher than the result calculated by the machine. Therefore, when the main memory module writes data, it will determine the priority of the writer (such as...). Figure 4b As shown in 430 (in the diagram), if the priority of the data to be written is lower than the priority of the current data writer, it is considered impossible to overwrite, and the process returns directly. After performing the priority comparison, the main memory module will update the storage based on this data and update the event table (e.g., ...). Figure 4bAs shown in 440), and snapshot table ( Figure 4b (As shown in 410). Figure 4b The "offline service Kafka" shown in section 450 refers to the fact that the main data storage module can provide offline services to users based on the Kafka message queue, such as using data for offline analysis; Figure 4b The "Kafka Online Service" shown in section 460 refers to the use of Kafka message queues to provide online services to users, such as online data analysis, which can provide real-time feedback of analysis results to users.

[0095] The primary storage database is mainly responsible for data writing. It is schema-less, writing data append-only, and is used to store and maintain change event logs and snapshots. Its simple logic, low error probability, and multiple replicas ensure high data reliability. Simultaneously, the primary storage database can generate data change streams through data comparison. The query module can subscribe to these change streams and synchronize them to the corresponding storage modules, ensuring data consistency between the primary storage database and the query modules. Within the primary storage database, the recorded change event logs allow users to retrieve snapshots of an article at any point in time; to retrieve change event records for all data within a specified time period of an article; and to roll back change event records for all data written to a specified time period and from a specified data writer. This historical data backtracking capability facilitates efficient business attribution and enables rapid data recovery.

[0096] S402: In response to a data service request from a requesting object, determine the data service type of the data service request.

[0097] Data service request types can include data analysis, data retrieval, data recommendation, and other service types, as well as content browsing, historical data query, and other service types.

[0098] S403: Determine the matching query module that matches the data service type from the configured multiple query modules. If the matching query module is a first type query module, then execute S404; if the matching query module is a second type query module, then execute S405.

[0099] The query module can synchronize data from the data change stream generated by the main data storage module. Different types of query modules can select appropriate storage media (such as Elasticsearch) to store the synchronized feature data in the corresponding storage module according to the functions and requirements of data queries in different scenarios. The query module can be a materialized read view built based on the data in the main data storage module. The first type of query module refers to a query module that needs to process feature data to provide services, and may include data analysis modules, data retrieval modules, and data recommendation modules. The second type of query module refers to a query module that does not need to process feature data to provide services, and may include content browsing modules and historical data query modules. The storage module corresponding to the first type of query module stores multiple media feature data groups. Each feature data group includes static and dynamic feature data of a media, and is determined based on the feature data change stream synchronized by the query module from the main data storage module.

[0100] S404: Call the first type of matching query module to obtain the matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module.

[0101] Optionally, if the data service type can be a data analysis service type, the matching query module matching this data service type can be a data analysis module, and the static feature data stored in the storage module corresponding to the data analysis module includes a media identifier. In response to the data service request, the data analysis module can be invoked to determine, from multiple media feature data groups stored in the storage module corresponding to the data analysis module, a first media feature data group containing a media identifier that matches the media identifier to be analyzed; matching media feature data matching the feature identifier to be analyzed is obtained from the first media feature data group; data analysis is performed on the matching media feature data to obtain the analysis result, and the analysis result is determined as the target data requested by the data service request.

[0102] Optionally, the data service type can be a data retrieval service type, and the matching query module matching this data service type can be a data retrieval module. The static feature data stored in the storage module corresponding to the data retrieval module includes media identifiers and media tags. The media tags can refer to the category of the media, such as sports news, entertainment news, etc. In response to the data service request, the data retrieval module is invoked to determine a second media feature data group containing media tags that match the search keywords from multiple media feature data groups stored in the storage module corresponding to the data retrieval module; the target media identifier contained in the second media feature data group is obtained, which is the matching media feature data that matches the data service request; the media content data corresponding to the target media identifier is obtained from the main data storage module, and the obtained media content data is determined as the target data requested by the data service request.

[0103] Optionally, the data service type can be a data recommendation service type, and the matching query module can be a data recommendation module. The static feature data stored in the storage module corresponding to the data recommendation module may include media identifiers, media tags, and publication time. In response to the data service request, the data recommendation module can be invoked to determine the media feature data groups that meet the filtering conditions based on the media tags (sports, entertainment, etc.) in the multiple media feature data groups stored in the storage module, and perform filtering processing on the media feature data groups that meet the filtering conditions. The data recommendation module can also determine the recommended media identifier from the unfiltered media feature data groups based on the media tags and publication time in the unfiltered media feature data groups stored in the storage module. The recommended media identifier is the matching media feature data (sports events, sports stars; entertainment award ceremonies, entertainment stars, etc.) that matches the data service request; retrieve the media content data corresponding to the recommended media identifier from the main data storage module, and determine the retrieved media content data as the target data requested by the data service request. Here, the filtering conditions may refer to the search keyword A involving illegal or unhealthy content. Filtering can involve deleting media feature data groups that meet the filtering criteria, or adding a filtering identifier to media feature data groups that meet the filtering criteria, so that when determining the target data, media feature data groups that meet the filtering criteria are excluded from the target data.

[0104] S405: Call the second type of matching query module to obtain the matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module.

[0105] Optionally, the data service type can be a content retrieval service type. If the data service request carries a media identifier to be retrieved, then the query module matching this data service request is the content retrieval module. The content retrieval module can synchronize media content data from the main data storage module and store the synchronized media content data in the corresponding cache module. In response to the data service request, the content retrieval module is invoked to query the media content data corresponding to the media identifier to be retrieved from the cache module; the media content data corresponding to the media identifier to be retrieved retrieved from the cache module is determined to be the target data requested by the data service request.

[0106] Because the cached data is time-sensitive, the cache window can be set according to different application scenarios. For example, only data within the last three months can be cached, while data older than three months can be cleared. In cases where the media content data corresponding to the media identifier to be queried cannot be found in the cache module, the media content data corresponding to the media identifier can be retrieved from the main memory database. The cache module differs from the storage module mentioned earlier; it can be a storage device built on Redis, thus supporting high-concurrency access to online content, with an average query latency of less than 10 milliseconds and a single node capable of supporting 100,000 query QPS.

[0107] Optionally, the data service type can be a historical data query service type. The data service request can carry the media identifier to be queried, the feature identifier to be queried, and the time to be queried. The query module matching this data service type is the historical data query module. In response to the data service request, the historical data query module is invoked to query the change event log of the media feature data corresponding to the media identifier to be queried from the main data storage module. Based on the queried change event log of the media feature data, the historical media feature data matching the feature identifier and the time to be queried is determined. The historical media feature data is determined as the target data requested by the data service request. Since the change event log stored in the main data storage module is stored in an append-only manner, there is no overwriting between the change event logs, so historical data at any recorded time point can be obtained.

[0108] The historical data query module enables online problem investigation and provides data rollback capabilities. For example, data processing systems often experience abnormal business performance at certain points in time. When analyzing the causes, it's necessary to reconstruct the situation at that time. However, because real-world business scenarios involve frequent updates, storing only the latest data means the original situation has been corrupted, making it impossible to reconstruct the original state and thus failing to meet the backtracking requirements. Furthermore, because the data production module has many data writers, in the context of rapid business iteration, some writers may inevitably write corrupted data to the main data storage module due to errors in their business programs. This necessitates that the main data storage module have data rollback capabilities. In such cases, calling the historical data query module can quickly remove corrupted data written by a specified writer, achieving the goal of rapid data rollback and quick recovery of business services.

[0109] In one implementation, the query module is flexibly extensible. New first-type or second-type query modules can also be constructed based on other business scenarios; this application does not limit this. For example, a time-series view can be constructed, which is a first-type query module. Its corresponding storage module can be used to collect all media content data and media characteristic data in chronological order, and data analysis can be performed based on the data stored in the storage module. For example, based on the historical changes in the readership of a news article, trend analysis can be performed to predict the future readership trend of that news article.

[0110] S406: Return the target data to the requesting object.

[0111] In this application, since the requesting objects have different query methods and format requirements for the target data, the matching query module can adapt and convert the data format according to the data business type of the requesting object to meet the data format requirements of different requesting objects.

[0112] For example, if the requesting object is a smartphone with a news app installed, and the requested data service type is historical data query, the data processing system can respond to the request, determine the target data, and then return the target data to the news app's display interface. For instance, a user can use the news app to search for a specific news article at any given time, including its text content, accompanying images, and characteristic data such as the number of views and reader comments up to that time.

[0113] It should be noted that, based on the same inventive concept, the technical details and principles of the data processing method in the above-mentioned manner can be found in the technical details and principles in S301-S304, which will not be repeated here for the sake of brevity.

[0114] The data processing method provided in this application is implemented based on a data processing system that includes a main data storage module and multiple query modules. The main data storage module is used to store media data and media feature data for various media, and to generate feature data change streams based on change events of the media feature data. Each query module is used to execute a type of data service, and the media feature data stored in the storage module corresponding to each query module is determined based on the feature data change stream synchronized by the query module from the main data storage module. Therefore, by constructing a main data storage module and configuring multiple query modules, this application adopts different storage schemes for the business data required by various data services, forming a Command Query Responsibility Segregation (CQRS) architecture. This enables the data processing system to respond to business requests from various data services and maintains stable service performance even under high concurrency. Furthermore, the query modules synchronize data through the data change stream of the main data storage module, which reduces data synchronization latency and ensures data consistency between the main data storage module and the query modules.

[0115] This application can determine the data service type of the data service request of the requesting object, and determine the matching query module that matches the data service type from multiple query modules configured in the data processing system. If the matching query module is a query module of the first type, it can be called to obtain the matching medium feature data that matches the service request from the corresponding storage module, and obtain the target data requested by the data service request based on the matching medium feature data, and return the target data to the requesting object. Therefore, this application allocates different query modules for different types of data service requests, which not only supports data querying for multiple types of data services, but also enables synchronous data querying for multiple types of data services through different query modules, thereby effectively improving the efficiency of data querying.

[0116] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 5 As shown, the data processing device includes a processing unit 510 and a transceiver unit 520, wherein:

[0117] Processing unit 510 is used to determine the data service type of the data service request in response to the data service request of the requesting object;

[0118] The processing unit 510 is used to determine the matching query module that matches the data service type from the multiple configured query modules; if the matching query module is a query module of the first type, the matching query module is called to obtain the matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and obtain the target data requested by the data service request based on the matching media feature data.

[0119] The transceiver unit 520 is used to return target data to the requesting object;

[0120] Each query module is used to execute one type of data service, and multiple query modules correspond to multiple types of data services. Multiple query modules are included in the data processing system, which also includes a data main storage module. The data main storage module is used to store media content data and media feature data, as well as to generate feature data change streams based on change events of media feature data. The media feature data stored in the storage module is determined based on the feature data change streams synchronized from the data main storage module by the query modules.

[0121] In one implementation, the storage module stores multiple media feature data groups, each media feature data group including static feature data and dynamic feature data of a medium; the static feature data includes a media identifier; the data service type is a data analysis service type, the data service request carries the feature identifier to be analyzed and the media identifier to be analyzed, the matching query module is a data analysis module used to perform data analysis, and the data analysis module is a query module of the first type; the processing unit 510 is also used to call the data analysis module, determine the first media feature data group whose media identifier matches the media identifier to be analyzed from the multiple media feature data groups stored in the storage module corresponding to the data analysis module; obtain the matching media feature data that matches the feature identifier to be analyzed from the first media feature data group; perform data analysis on the matching media feature data to obtain the analysis result, and determine the analysis result as the target data requested by the data service request.

[0122] In one implementation, the storage module stores multiple media feature data groups, each including static and dynamic feature data of a medium; the static feature data includes a media identifier and a media tag; the data service type is a data retrieval service type, the data service request carries search keywords, and the matching query module is a data retrieval module used to perform data retrieval, which is a first type of query module; the processing unit 510 is also used to call the data retrieval module to determine a second media feature data group whose media tags match the search keywords from the multiple media feature data groups stored in the storage module corresponding to the data retrieval module; obtain the target media identifier contained in the second media feature data group, the target media identifier being the matching media feature data that matches the data service request; obtain the media content data corresponding to the target media identifier from the main data storage module, and determine the obtained media content data as the target data requested by the data service request.

[0123] In one implementation, the storage module stores multiple media feature data groups, each including static and dynamic feature data of a media; the static feature data includes media identifier, media tag, and release time; the data service type is a data recommendation service type, and the matching query module is a data recommendation module used to perform data recommendation, which is a query module of the first type; the processing unit 510 is also used to call the data recommendation module, based on the media tag in the multiple media feature data groups stored in the storage module, to determine the media feature data groups that meet the filtering conditions, and to perform filtering processing on the media feature data groups that meet the filtering conditions; and the processing unit 510 is also used to call the data recommendation module, based on the media tag and release time in the unfiltered media feature data groups stored in the storage module, to determine the recommended media identifier from the unfiltered media feature data groups, the recommended media identifier being the matching media feature data that matches the data service request; and to obtain the media content data corresponding to the recommended media identifier from the main data storage module, and to determine the obtained media content data as the target data requested by the data service request.

[0124] In one implementation, the data service type is a content retrieval service type, and the data service request carries the identifier of the media to be retrieval; the matching query module is a content retrieval module used to perform content retrieval, and the content retrieval module is a second type of query module. The content retrieval module is used to synchronize media content data from the main data storage module and store the synchronized media content data in the cache module corresponding to the content retrieval module; then the processing unit 510 is also used to call the content retrieval module to query the media content data corresponding to the identifier of the media to be retrieval from the cache module; and determine the media content data corresponding to the identifier of the media to be retrieval retrieved from the cache module as the target data requested by the data service request.

[0125] In one implementation, the main data storage module is also used to store change event logs of media characteristic data for each medium. The change event logs are stored in an append-only manner. The data service type is a historical data query service type. The data service request carries the media identifier to be queried, the feature identifier to be queried, and the query time. The matching query module is a historical data query module used to perform historical data queries. The historical data query module is a second type of query module. The processing unit 510 is also used to call the historical data lookup module to query the change event logs of the media characteristic data corresponding to the media identifier to be queried from the main data storage module. Based on the queried change event logs of the media characteristic data, the historical media characteristic data that matches the feature identifier and the query time is determined. The historical media characteristic data is determined as the target data requested by the data service request.

[0126] In one implementation, the main data storage module is used to store the static and dynamic characteristic data of the medium. The static characteristic data of the medium is stored in the main memory database of the main data storage module. The main data storage module also includes a dynamic characteristic unit, which is used to sample changes in the dynamic characteristic data of the medium and store the sampled changed dynamic characteristic data into the main memory database. The characteristic data change stream is generated based on the medium characteristic data before the change and the obtained changed medium characteristic data.

[0127] According to one embodiment of this application, Figure 3a and Figure 4a The steps involved in the data processing method shown can be derived from... Figure 5 The data processing device shown is executed by various modules. For example, Figure 3a The steps S301 and S301 shown are as follows: Figure 4a Step S402 shown can be performed by, for example Figure 5 The processing unit 510 shown is used to execute this; Figure 3a The steps S302, S303 and shown are Figure 4a Steps S401, S403-S405 shown can be derived from, for example: Figure 5 The processing unit 510 shown is used to execute this; Figure 3a Step S304 and shown Figure 4a Step S406 shown can be performed by, for example Figure 5 The transceiver unit 520 shown is used to perform this operation.

[0128] According to one embodiment of this application, Figure 5The modules in the data processing apparatus shown can be individually or entirely combined into one or more units, or some of these units can be further divided into multiple functionally smaller sub-units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical functional division. In practical applications, the function of one module can be implemented by multiple units, or the function of multiple modules can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0129] It is understood that the functions of each functional unit of the data processing device described in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0130] As can be seen, this application allocates different query modules for data querying based on different types of data service requests. This not only supports data querying for multiple types of data services, but also enables synchronous data querying for multiple types of data services through different query modules, thereby effectively improving the efficiency of data querying.

[0131] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 6 As shown in the embodiments of this application, the computer device described is used to execute the data processing method described above, and includes: a processor 610, a communication interface 620, and a memory 630. The processor 610, communication interface 620, and memory 630 can be connected via a bus or other means; this embodiment of the application takes a bus connection as an example.

[0132] The processor 610 (or CPU, Central Processing Unit) is the computing and control core of the computer device. It can parse various instructions and process various data within the computer device. For example, the CPU can parse power-on / off commands sent to the computer device and control the device to perform power-on / off operations; it can also transmit various interactive data between internal structures of the computer device. The communication interface 620 may optionally include standard wired interfaces or wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), controlled by the processor 610 for sending and receiving data. The memory 630 is a storage device in the computer device used to store programs and data. It is understood that the memory 630 here can include the computer device's built-in memory, or it can include extended memory supported by the computer device. The memory 630 provides storage space for the computer device's operating system, which may include, but is not limited to, Android, iOS, Windows Phone, etc., and this application does not limit this.

[0133] In this embodiment of the application, the processor 610 performs the following operations by running the executable program code in the memory 630:

[0134] In response to a data service request from a requesting object, determine the data service type of the data service request;

[0135] The matching query module that matches the data service type is determined from the multiple configured query modules; if the matching query module is the first type of query module, the matching query module is called to obtain the matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and the target data requested by the data service request is obtained based on the matching media feature data.

[0136] Return the target data to the requested object;

[0137] Each query module is used to execute one type of data service, and multiple query modules correspond to multiple types of data services. Multiple query modules are included in the data processing system, which also includes a data main storage module. The data main storage module is used to store media content data and media feature data, as well as to generate feature data change streams based on change events of media feature data. The media feature data stored in the storage module is determined based on the feature data change streams synchronized from the data main storage module by the query modules.

[0138] In one implementation, the storage module stores multiple media feature data groups, each media feature data group including static feature data and dynamic feature data of a medium; the static feature data includes a media identifier; the data service type is a data analysis service type, the data service request carries the feature identifier to be analyzed and the media identifier to be analyzed, the matching query module is a data analysis module used to perform data analysis, and the data analysis module is a query module of the first type; the processor 610 can also perform the following operations by running the executable program code in the memory 630:

[0139] The data analysis module is invoked to determine the first media feature data group from the multiple media feature data groups stored in the storage module corresponding to the data analysis module. The first media feature data group is used to obtain the matching media feature data that matches the feature identifier to be analyzed. The matching media feature data is then analyzed to obtain the analysis results, and the analysis results are determined as the target data requested by the data service request.

[0140] In one implementation, the storage module stores multiple media feature data groups, each media feature data group including static feature data and dynamic feature data of a media; the static feature data includes media identifier and media tag; the data service type is a data retrieval service type, the data service request carries retrieval keywords, the matching query module is a data retrieval module used to perform data retrieval, and the data retrieval module is a query module of the first type; the processor 610 can also perform the following operations by running the executable program code in the memory 630:

[0141] The data retrieval module is invoked to determine the second media feature data group whose media tags match the search keywords from multiple media feature data groups stored in the storage module corresponding to the data retrieval module; the target media identifier contained in the second media feature data group is obtained, and the target media identifier is the matching media feature data that matches the data service request; the media content data corresponding to the target media identifier is obtained from the main data storage module, and the obtained media content data is determined as the target data requested by the data service request.

[0142] In one implementation, the storage module stores multiple media feature data groups, each media feature data group including static feature data and dynamic feature data of a media; the static feature data includes media identifier, media tag, and release time; the data service type is a data recommendation service type, and the matching query module is a data recommendation module used to perform data recommendation, which is a query module of the first type; the processor 610 can also perform the following operations by running the executable program code in the memory 630:

[0143] Based on the media tags in multiple media feature data groups stored in the storage module, the system identifies media feature data groups that meet the filtering conditions and performs filtering processing on these groups. Additionally, it calls the data recommendation module to determine recommended media identifiers from the unfiltered media feature data groups stored in the storage module, based on the media tags and release times. These recommended media identifiers are the matching media feature data that matches the data service request. Finally, it retrieves the media content data corresponding to the recommended media identifiers from the main data storage module and identifies the retrieved media content data as the target data requested by the data service request.

[0144] In one implementation, the data service type is a content retrieval service type, and the data service request carries the identifier of the media to be retrieved; the matching query module is a content retrieval module used to perform content retrieval, and the content retrieval module is a second type of query module. The content retrieval module is used to synchronize media content data from the main data storage module and store the synchronized media content data in the cache module corresponding to the content retrieval module; the processor 610 can also perform the following operations by running the executable program code in the memory 630:

[0145] The content retrieval module is invoked to query the media content data corresponding to the media identifier to be retrieval from the cache module; the media content data corresponding to the media identifier to be retrieval from the cache module is determined to be the target data requested by the data service request.

[0146] In one implementation, the main data memory module is also used to store change event logs of media characteristic data for each medium. These change event logs are stored in an append-only manner. The data service type is a historical data query service type. The data service request carries the media identifier to be queried, the characteristic identifier to be queried, and the time to be queried. The matching query module is a historical data query module used to execute historical data queries, and this historical data query module is a second type of query module. Then, the processor 610, by running the executable program code in the memory 630, can also perform the following operations:

[0147] The historical data retrieval module is invoked to query the change event log of the media feature data corresponding to the media identifier to be queried from the main data storage module; based on the queried change event log of the media feature data, the historical media feature data that matches the feature identifier and the time to be queried is determined; the historical media feature data is determined as the target data requested by the data service request.

[0148] In one implementation, the main data storage module is used to store the static and dynamic characteristic data of the medium. The static characteristic data of the medium is stored in the main memory database of the main data storage module. The main data storage module also includes a dynamic characteristic unit, which is used to sample changes in the dynamic characteristic data of the medium and store the sampled changed dynamic characteristic data into the main memory database. The characteristic data change stream is generated based on the medium characteristic data before the change and the obtained changed medium characteristic data.

[0149] As can be seen, this application allocates different query modules for data querying based on different types of data service requests. This not only supports data querying for multiple types of data services, but also enables synchronous data querying for multiple types of data services through different query modules, thereby effectively improving the efficiency of data querying.

[0150] This application also provides a computer-readable storage medium storing a computer program executed by the aforementioned data processing apparatus, the computer program including program instructions, which, when executed by a processor, enable the execution of the aforementioned data processing apparatus. Figure 3a and Figure 4a The description of the data processing method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0151] As an example, the above program instructions can be deployed and executed on a single computer device, or deployed and executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. Multiple computer devices distributed across multiple locations and interconnected via a communication network can form a blockchain network.

[0152] The aforementioned computer-readable storage medium can be an internal storage unit of the data processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0153] This application provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned... Figure 3a and Figure 4a The data processing method described in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0154] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0155] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0156] The methods and related apparatuses provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0157] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: In response to a data service request from a requesting object, determine the data service type of the data service request; From the configured multiple query modules, determine the matching query module that matches the data service type; If the matching query module is a first type of query module, then the matching query module is invoked to obtain matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and the target data requested by the data service request is obtained based on the matching media feature data. Return the target data to the requested object; Each query module is used to execute one type of data service, and the multiple query modules correspond to multiple types of data services. The multiple query modules are included in a data processing system, which also includes a data main storage module. The data main storage module is used to store media content data and media feature data of the media, and to generate feature data change streams based on change events of the media feature data. The media feature data stored in the storage module is determined based on the feature data change streams synchronized by the query modules from the data main storage module. The feature data change streams of the same media are processed serially by the same processing coroutine in the same query module, so that the update order of the media feature data in the storage module is consistent with the change order in the data main storage module. Each query module's corresponding storage module adopts a data storage method that matches the data service type executed by the query module; the data processing system adopts a Command Query Responsibility Separation (CQRS) architecture, where the main data storage module acts as the command end, responsible for data writing and change stream generation, and the multiple query modules act as query ends, responsible for processing query requests for their respective data service types.

2. The method as described in claim 1, characterized in that, The storage module stores multiple media feature data groups, each media feature data group including static feature data and dynamic feature data of a medium, the static feature data including a media identifier; the data service type is a data analysis service type, the data service request carries the feature identifier to be analyzed and the media identifier to be analyzed, the matching query module is a data analysis module for performing data analysis, and the data analysis module is a query module of the first type; The step of calling the matching query module to obtain matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and obtaining the target data requested by the data service request based on the matching media feature data, includes: The data analysis module is invoked to determine, from the multiple media feature data groups stored in the storage module corresponding to the data analysis module, the first media feature data group whose media identifier matches the media identifier to be analyzed. Obtain matching media feature data that matches the feature identifier to be analyzed from the first media feature data group; The matching medium feature data is analyzed to obtain the analysis results, and the analysis results are determined as the target data requested by the data service request.

3. The method as described in claim 1, characterized in that, The storage module stores multiple media feature data groups, each media feature data group including static feature data and dynamic feature data of a medium, the static feature data including media identifier and media tag; the data service type is a data retrieval service type, the data service request carries retrieval keywords, the matching query module is a data retrieval module used to perform data retrieval, and the data retrieval module is the query module of the first type; The step of calling the matching query module to obtain matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and obtaining the target data requested by the data service request based on the matching media feature data, includes: The data retrieval module is invoked to determine the second media feature data group whose media tags match the search keywords from multiple media feature data groups stored in the storage module corresponding to the data retrieval module. Obtain the target media identifier contained in the second media feature data group, wherein the target media identifier is the matching media feature data that matches the data service request; The media content data corresponding to the target media identifier is obtained from the main data storage module, and the obtained media content data is determined as the target data requested by the data service request.

4. The method as described in claim 1, characterized in that, The storage module stores multiple media feature data groups, each media feature data group including static feature data and dynamic feature data of a media. The static feature data includes media identifier, media tag, and release time. The data service type is a data recommendation service type, and the matching query module is a data recommendation module used to perform data recommendation. The data recommendation module is a query module of the first type. The method further includes: The data recommendation module is invoked, and based on the media tags in the multiple media feature data groups stored in the storage module, the media feature data groups that meet the filtering conditions are determined. The media feature data groups that meet the filtering conditions are filtered. The step of calling the matching query module to obtain matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and obtaining the target data requested by the data service request based on the matching media feature data, includes: The data recommendation module is invoked to determine a recommended media identifier from the unfiltered media feature data group stored in the storage module, based on the media tags and publication time. The recommended media identifier is the matching media feature data that matches the data service request. The media content data corresponding to the recommended media identifier is obtained from the main data storage module, and the obtained media content data is determined as the target data requested by the data service request.

5. The method as described in claim 1, characterized in that, The data service type is a content retrieval service type, and the data service request carries the identifier of the media to be retrieval; the matching query module is a content retrieval module for performing content retrieval, and the content retrieval module is a second type of query module. The content retrieval module is used to synchronize media content data from the main data storage module and store the synchronized media content data in the cache module corresponding to the content retrieval module; the method further includes: The content retrieval module is invoked to query the media content data corresponding to the media identifier to be retrieval from the cache module; The media content data corresponding to the media identifier to be viewed will be retrieved from the cache module and determined to be the target data requested by the data service request.

6. The method as described in claim 1, characterized in that, The main data storage module is also used to store change event logs of media characteristic data for each medium, and the change event logs are stored in an append-only manner; the data service type is a historical data query service type, and the data service request carries the media identifier to be queried, the characteristic identifier to be queried, and the time to be queried; the matching query module is a historical data query module used to perform historical data queries, and the historical data query module is a second type of query module; the method further includes: Call the historical data query module to query the change event log of the media feature data corresponding to the media identifier to be queried from the data main storage module; Based on the change event log of the queried media feature data, determine the historical media feature data that matches the feature identifier to be queried and the time to be queried; The historical media feature data is identified as the target data requested by the data service request.

7. The method according to any one of claims 1-6, characterized in that, The main data storage module is used to store the static and dynamic feature data of the medium. The static feature data of the medium is stored in the main memory database of the main data storage module. The main data storage module also includes a dynamic feature unit, which is used to sample changes in the dynamic feature data of the medium and store the sampled changed dynamic feature data into the main memory database. The feature data change stream is generated based on the media feature data before the change and the obtained changed media feature data.

8. A data processing apparatus, characterized in that, include: The processing unit is configured to determine the data service type of the data service request in response to the data service request of the requesting object; The processing unit is configured to determine a matching query module that matches the data service type from a plurality of configured query modules; if the matching query module is a query module of the first type, the matching query module is invoked to obtain matching media feature data that matches the data service request from the media feature data stored in the storage module corresponding to the matching query module, and the target data requested by the data service request is obtained based on the matching media feature data. The transceiver unit is used to return the target data to the requesting object; Each query module is used to execute one type of data service, and the multiple query modules correspond to multiple types of data services; the multiple query modules are included in the data processing system, and the data processing system also includes a data main storage module; the data main storage module is used to store media content data and media feature data of the media, and to generate feature data change streams based on change events of media feature data; The media feature data stored in the storage module is determined based on the feature data change stream synchronized from the main data storage module by the query module. The feature data change stream of the same medium is processed serially by the same processing coroutine in the same query module, so that the update order of the media feature data in the storage module is consistent with the change order in the main data storage module. Each query module's corresponding storage module adopts a data storage method that matches the data service type executed by the query module; the data processing system adopts a Command Query Responsibility Separation (CQRS) architecture, where the main data storage module acts as the command end, responsible for data writing and change stream generation, and the multiple query modules act as query ends, responsible for processing query requests for their respective data service types.

9. A computer device, characterized in that, The computer device includes a processor, a communication interface, and a memory, wherein the processor, the communication interface, and the memory are interconnected, and the memory stores executable program code. The processor is used to call the executable program code to implement the data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed by a processor, is used to implement the data processing method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer storage medium, which, when executed by a processor, is used to implement the data processing method as described in any one of claims 1-7.

Images