Data processing method and device, equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, equipment, a storage medium and a program product, and can be applied to the field of storage systems and data management. The data processing method comprises the steps that input data are matched with multiple data types in a data type library, multiple target modes corresponding to the input data are obtained, and the multiple data types are stored in the data type library; based on the target mode, performing fragmentation polling processing on the input data to obtain target address information corresponding to the target mode, the fragmentation polling processing including segmenting the input data into a plurality of pieces of target mode data and processing the plurality of pieces of target mode data by using a plurality of threads; and storing a target mapping relationship between the multiple target modes of the input data and the corresponding target address information into a data type library.

Description

Technical Field

[0001] This application relates to the field of storage systems and data management, and more specifically to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Data deduplication reduces system redundancy and saves space by keeping a copy of data blocks with identical content and replacing the remaining copies with pointers. Schema-based data deduplication directly models repeatedly stored data blocks as schemas, without storing the actual data. However, related pattern recognition algorithms can only recognize constant patterns (all 0s or all 1s), and cannot model periodically changing data, forcing them to be written as is, thus reducing deduplication efficiency and failing to improve storage space utilization. Summary of the Invention

[0003] In view of the above problems, this application provides a data processing method, apparatus, device, storage medium and program product.

[0004] According to a first aspect of this application, a data processing method is provided, comprising: matching input data with multiple data types in a data type library to obtain multiple target patterns corresponding to each input data, wherein the data type library stores multiple data types; performing slice polling processing on the input data based on the target patterns to obtain target address information corresponding to the target patterns, wherein the slice polling processing includes splitting the input data into multiple target pattern data and processing the multiple target pattern data using multiple threads; and storing the target mapping relationship between the multiple target patterns of the input data and the corresponding target address information in the data type library.

[0005] According to embodiments of this application, by matching input data with multiple data types in a data type library, a target pattern can be determined, expanding from traditional 0 / 1 binary data to multiple data types. This achieves refined classification of multiple pattern types and improves data storage efficiency. Based on the target pattern, input data can be segmented into multiple target pattern data. Multiple threads can be used to process multiple target pattern data, dynamically balancing the computational load. This solves the problem of single-core CPU overload caused by continuous input of a single data stream, making CPU utilization more balanced. Finally, the target mapping relationship between the multiple target patterns of the input data and the target address information corresponding to the target pattern is stored in the data type library to improve access efficiency during subsequent data access.

[0006] According to an embodiment of this application, the input data is matched with multiple data types in a data type library to obtain multiple target patterns corresponding to each input data, including: performing fingerprint matching between the input data and multiple data types to obtain a target data type; and comparing the input data with multiple patterns included in the target data type to determine the target pattern of the input data.

[0007] According to the embodiments of this application, by extending from traditional 0 / 1 binary data to multiple data types, a refined classification of multiple types of patterns is achieved, improving the matching rate of input data and increasing the system reduction rate. In addition, the type fingerprint is calculated once and resides in memory, and comparison can be performed using registers, reducing CPU overhead.

[0008] According to embodiments of this application, comparing input data with multiple patterns included in the target data type to determine the target pattern of the input data includes: when the target data type is numeric, comparing the input data with multiple periodic numeric patterns included in the numeric type to determine the target pattern of the input data; when the target data type is multimedia, comparing the input data with multiple video frame patterns included in the multimedia type to determine the target pattern of the input data, wherein the video frame patterns include a full black frame pattern and a full white frame pattern; when the target data type is sampling, comparing the input data with multiple sampling patterns included in the sampling type to determine the target pattern of the input data, wherein the sampling patterns include a constant sampling pattern and a periodic sampling pattern; and when the target data type is log, comparing the input data with a template log pattern included in the log type to determine the target pattern of the input data.

[0009] According to embodiments of this application, when the target data type is numerical, the input data can be compared with multiple periodic numerical patterns, expanding from only recognizing 0 / 1 binary data to periodic data, thus improving the efficiency of data type matching. When the target data type is multimedia, the input data can be compared with multiple video frame patterns. By extracting the common parts of the multimedia types, video frame patterns are obtained, such as all-black frames or all-white frames. By identifying the common video frames as pattern data, the utilization rate of storage space is improved. When the target data type is sampling, the general features of sampling can be classified based on dimensions such as time or space, and the general features are identified as pattern data, thereby achieving refined classification of multiple data patterns. When the target data type is log, the template log can be identified as a pattern. Therefore, when the input data is log data, the input data can be divided into pattern data and non-pattern data. The template logs of multiple logs are extracted, and the non-pattern data is saved separately, thereby improving the utilization rate of storage space.

[0010] According to an embodiment of this application, the data processing method further includes: inputting pattern data that meets a preset number of hits into a target pattern prediction model, outputting predicted pattern data, and writing the predicted pattern data into a data type library.

[0011] According to embodiments of this application, pattern data with a high hit rate can be processed by a target pattern prediction model to obtain predicted pattern data. By expanding the data type library, the matching probability of input data with multiple data types can be increased, invalid storage operations can be reduced, and fine classification of multiple data patterns can be achieved. By writing the predicted pattern data in advance, storage efficiency can be improved.

[0012] According to embodiments of this application, the target pattern prediction model includes at least one of the following: a first target pattern prediction model for numerical prediction, a second target pattern prediction model for multimedia prediction, a third target pattern prediction model for sampling prediction, and a fourth target pattern prediction model for log prediction; wherein, the first target pattern prediction model is trained using periodic numerical fingerprint data and target pattern labels corresponding to the periodic numerical data, the periodic numerical fingerprint data being obtained by processing periodic numerical data based on an initial first target pattern prediction model; the second target pattern prediction model is trained using video frame fingerprint data and video frame pattern labels corresponding to the video frame data, the video frame fingerprint data being obtained by processing video frame data based on an initial second target pattern prediction model; the third target pattern prediction model is trained using sampled fingerprint data and sampled pattern labels corresponding to the sampled data, the sampled fingerprint data being obtained by processing sampled data based on an initial third target pattern prediction model; and the fourth target pattern prediction model is trained using template log fingerprint data and template log pattern labels corresponding to the template log data, the template log fingerprint data being obtained by processing template log data based on an initial fourth target pattern prediction model.

[0013] According to embodiments of this application, for each data type, a corresponding target pattern prediction model can be trained. By processing the input data using the target pattern prediction model, predicted pattern data can be obtained. By expanding the matching data types, storage efficiency and computing resource utilization can be improved.

[0014] According to an embodiment of this application, the data processing method further includes: in response to receiving a read request, determining pattern data corresponding to the read request based on a target mapping relationship; and determining the data to be read based on the pattern data.

[0015] According to the embodiments of this application, the data segment can be determined as pattern data through the target mapping relationship. Target address information corresponding to multiple pattern data can be obtained, saving disk I / O resources. In addition, the corresponding pattern data can be obtained based on the target address information. By combining multiple pattern data, the data to be read can be obtained. Periodic or constant patterns are virtualized, and the physical space can store non-pattern data and metadata, reducing storage costs.

[0016] A second aspect of this application provides a data processing apparatus, comprising: a matching module for matching input data with multiple data types in a data type library to obtain multiple target patterns corresponding to each input data, the data type library storing multiple data types; a slice polling module for performing slice polling processing on the input data based on the target patterns to obtain target address information corresponding to the target patterns, the slice polling processing including splitting the input data into multiple target pattern data and processing the multiple target pattern data using multiple threads; and a storage module for storing the target mapping relationship between the multiple target patterns of the input data and the corresponding target address information in the data type library.

[0017] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0018] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0019] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0020] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0021] Figure 1 The illustration shows application scenario diagrams of data processing methods, apparatus, devices, media, and program products according to embodiments of this application.

[0022] Figure 2 A flowchart of a data processing method according to an embodiment of this application is shown.

[0023] Figure 3 An architecture diagram of a target pattern prediction model according to an embodiment of this application is shown.

[0024] Figure 4 An architecture diagram of a data processing method according to an embodiment of this application is shown.

[0025] Figure 5 A flowchart of a data processing method according to another embodiment of this application is shown.

[0026] Figure 6 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown.

[0027] Figure 7 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of this application is shown. Detailed Implementation

[0028] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] Data deduplication is a data compression technique that reduces redundant data in storage systems by retaining only unique copies of identical data. In modern environments, data duplication is extremely common. For example, multiple copies of files, redundant messages in email systems, and multiple data uploads can all lead to a large amount of duplicate data in storage systems. This duplicate data not only occupies a significant amount of storage space but also increases the burden on data transfer, backup, and processing within the system.

[0033] Data deduplication typically uses hash algorithms to calculate data fingerprints. Data with the same fingerprints are compared byte by byte to find identical data blocks. The mapping relationship between the new data block and the existing data is then established, thereby achieving the purpose of deduplication. Data deduplication technology is widely used in various scenarios, such as backup storage, virtualization environments, cloud storage services, and archiving systems.

[0034] A pattern data type refers to a block of data consisting entirely of zeros or all ones, such as all zeros (0x00000...) or all ones (0xFFFFFF...). This type of data exhibits a high degree of regularity. If a 1MB block of data all belongs to the pattern data, only a few bytes of tags are needed to store the entire 1MB of data. During data reduction, once pattern data is identified, only metadata is added; the data is not flushed to disk. On the next read, if the pattern data tag is found, it will be directly returned to the user. Pattern technology can reduce disk space and shorten I / O paths.

[0035] However, current pattern-based deduplication techniques can only recognize numeric data consisting entirely of 0s and 1s, which constitute a very small percentage of the overall storage system. They also fail to recognize data with regular, periodic changes (such as 0xabcabcabc). Furthermore, frequently printed, identical log data in log storage can also be approximated as pattern data. None of these data types can be recognized by the relevant pattern model.

[0036] Furthermore, if the same pattern data appears frequently, the storage system will assign it to a fixed thread for processing because the fingerprints are the same, which will cause excessive pressure on a single central processing unit (CPU).

[0037] The existing pattern technology is not intelligent enough and cannot be extended based on existing data. Some time-series data, such as temperature and humidity, have simple values ​​and are somewhat predictable, but current pattern technology cannot process them intelligently. For multimedia data such as video, large-scale fingerprint calculations and data feature collection are not possible, nor can predictions or extensions be performed.

[0038] To address at least one of the aforementioned problems, embodiments of this application provide a data processing method that matches input data with multiple data types in a data type library to obtain multiple target patterns corresponding to each input data type. The data type library stores multiple data types. Based on the target patterns, the input data is processed by segmentation and polling to obtain target address information corresponding to the target patterns. The segmentation and polling process includes dividing the input data into multiple target pattern data and processing multiple target pattern data using multiple threads. The target mapping relationship between the multiple target patterns of the input data and the corresponding target address information is stored in the data type library.

[0039] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0040] Figure 1 The illustration shows application scenario diagrams of data processing methods, apparatus, devices, media, and program products according to embodiments of this application.

[0041] like Figure 1 As shown, the application scenario according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a data type library 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the data type library 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0042] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the data type library 105 via the network 104 to receive or send messages, etc. Various communication terminal applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0043] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays that support browsing or uploading, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0044] The data type library 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0045] It should be noted that the data processing method provided in the embodiments of this application can generally be executed by the data type library 105. Correspondingly, the data processing device provided in the embodiments of this application can generally be located in the data type library 105. The data processing method provided in the embodiments of this application can also be executed by a server or server cluster that is different from the data type library 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the data type library 105. Correspondingly, the data processing device provided in the embodiments of this application can also be located in a server or server cluster that is different from the data type library 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the data type library 105.

[0046] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0047] The following will be based on Figure 1 The described scene, through Figures 2-5 The data processing method according to the embodiments of this application will be described in detail.

[0048] Figure 2 A flowchart of a data processing method according to an embodiment of this application is shown.

[0049] like Figure 2As shown, the data processing method of this embodiment includes operations S210 to S230.

[0050] In operation S210, the input data is matched with various data types in the data type library to obtain various target patterns corresponding to each input data.

[0051] During operation S220, based on the target pattern, the input data is processed by fragmented polling to obtain the target address information corresponding to the target pattern.

[0052] In operation S230, the target mapping relationship between the various target modes of the input data and the corresponding target address information is stored in the data type library.

[0053] The data type library allows for the preset setting of various data types, including numeric, multimedia, sampling, log, and other types. Other types can represent data that was not matched. Each data type can include multiple patterns. Input data can be matched against the various patterns corresponding to each data type in the data type library to determine the target pattern. For example, input data is 012345.....012345. It can be matched against multiple data types to determine that the input data is a numeric data type. Then, the input data is further compared with the various periodic numeric patterns included in the numeric type to finally determine the target pattern, which is "012345".

[0054] Sharded polling processing can include splitting input data into multiple target pattern data and using multiple threads to process multiple target pattern data. When the input data volume exceeds a preset limit, it can be divided into multiple target pattern data. For example, in the case of large amounts of numerical data, it can be divided into multiple target pattern data based on a target pattern (i.e., periodic values). By dividing large amounts of data into smaller amounts, it prepares for subsequent processing. Polling the divided target pattern data among various worker threads ensures that the CPU load is as consistent as possible, achieving dynamic load balancing.

[0055] The target mapping relationship between multiple target patterns of input data and their corresponding target address information can be stored in a data type library. The target mapping relationship can be [key, value]. The key can represent the target pattern, and the value can represent the target address information corresponding to that target pattern. After identifying the target pattern data, metadata can be added, preventing the data from being flushed to disk. During the next read, if the target pattern tag is found, it will be directly returned to the user, thereby reducing the duplication rate of disk space and shortening the IO path.

[0056] According to embodiments of this application, by matching input data with multiple data types in a data type library, a target pattern can be determined, expanding from traditional 0 / 1 binary data to multiple data types. This achieves refined classification of multiple pattern types and improves data storage efficiency. Based on the target pattern, input data can be segmented into multiple target pattern data. Multiple threads can be used to process multiple target pattern data, dynamically balancing the computational load. This solves the problem of single-core CPU overload caused by continuous input of a single data stream, making CPU utilization more balanced. Finally, the target mapping relationship between the multiple target patterns of the input data and the target address information corresponding to the target pattern is stored in the data type library to improve access efficiency during subsequent data access.

[0057] According to an embodiment of this application, the input data is matched with multiple data types in a data type library to obtain multiple target patterns corresponding to each input data, including: performing fingerprint matching between the input data and multiple data types to obtain a target data type; and comparing the input data with multiple patterns included in the target data type to determine the target pattern of the input data.

[0058] Data types can include numeric, multimedia, sampled, log, and other types. Numeric types can contain various periodic numerical patterns, such as all 0s and all 1s, as well as patterns that appear at fixed intervals, such as 0xaaa…aaa, 0xabcabc, etc. Multimedia types can include music or video frame patterns that repeat multiple times. Sampled types can represent common characteristic patterns that can be categorized or exhaustively enumerated according to time, space, or a certain dimension. For each type of log data, with only minor differences, such as different log return values ​​or thread numbers, the different data can be stored directly on disk, while the common data can be used as a template log pattern.

[0059] A type fingerprint can be pre-calculated for each data type (e.g., all 0s, all 1s, incrementing, decrementing, etc.). Input data can be compared with these type fingerprints in parallel, and once a match is found, the input data is marked as the corresponding target data type. By quickly converging the input data to a finite number of known types, extensive subsequent traversal is avoided. The input data can be compared byte-by-byte with multiple patterns included in the target data type to determine the target pattern of the input data.

[0060] According to the embodiments of this application, by extending from traditional 0 / 1 binary data to multiple data types, a refined classification of multiple types of patterns is achieved, improving the matching rate of input data and increasing the system reduction rate. In addition, the type fingerprint is calculated once and resides in memory, and comparison can be performed using registers, reducing CPU overhead.

[0061] According to embodiments of this application, comparing input data with multiple patterns included in the target data type to determine the target pattern of the input data includes: when the target data type is numeric, comparing the input data with multiple periodic numeric patterns included in the numeric type to determine the target pattern of the input data; when the target data type is multimedia, comparing the input data with multiple video frame patterns included in the multimedia type to determine the target pattern of the input data, wherein the video frame patterns include a full black frame pattern and a full white frame pattern; when the target data type is sampling, comparing the input data with multiple sampling patterns included in the sampling type to determine the target pattern of the input data, wherein the sampling patterns include a constant sampling pattern and a periodic sampling pattern; and when the target data type is log, comparing the input data with a template log pattern included in the log type to determine the target pattern of the input data.

[0062] When the target data type is numeric, the input data can be compared with various periodic numeric patterns contained within the numeric type to determine the target pattern of the input data. Numeric types can include multiple periodic numeric patterns, each with the same fingerprint. Each periodic numeric pattern also includes multiple numeric patterns. For example, the numeric pattern "abdabdabdabd" has the same fingerprint as the numeric pattern "abdbad," both belonging to the same periodic numeric pattern. However, specifically, a byte-by-byte comparison is required to determine the final target pattern.

[0063] When the target data type is multimedia, the input data can be compared with various video frame modes included in the multimedia type to determine the target mode of the input data. For example, for movie video data, video frame modes may include all-black frame mode, all-white frame mode, copy protection watermark information, copyright notice frames, and uniformly released movie trailers, etc. When the input data is a specific movie, the input data can be compared with various video frame modes included in the multimedia type to determine the target mode of the input data.

[0064] When the target data type is sampling-based, the input data is compared with various sampling patterns included in the sampling type to determine the target pattern of the input data. Sampling patterns can include constant sampling patterns and periodic sampling patterns. For example, traffic flow can be sampled; for peak periods, a first sampling frequency value can be determined, and for off-peak periods, a second sampling frequency can be determined. Pattern data can represent the first and second sampling frequency values. A constant sampling pattern indicates a constant sampling frequency, and this constant sampling frequency can be defined as pattern data.

[0065] When the target data type is log, the input data is compared with the template log patterns included in the log type to determine the target pattern of the input data. For example, for weather log data, the log data may include template logs and corresponding weather data, with only some differences. The input log data can be compared with multiple template log patterns to determine the target pattern.

[0066] According to embodiments of this application, when the target data type is numerical, the input data can be compared with multiple periodic numerical patterns, expanding from only recognizing 0 / 1 binary data to periodic data, thus improving the efficiency of data type matching. When the target data type is multimedia, the input data can be compared with multiple video frame patterns. By extracting the common parts of the multimedia types, video frame patterns are obtained, such as all-black frames or all-white frames. By identifying the common video frames as pattern data, the utilization rate of storage space is improved. When the target data type is sampling, the general features of sampling can be classified based on dimensions such as time or space, and the general features are identified as pattern data, thereby achieving refined classification of multiple data patterns. When the target data type is log, the template log can be identified as a pattern. Therefore, when the input data is log data, the input data can be divided into pattern data and non-pattern data. The template logs of multiple logs are extracted, and the non-pattern data is saved separately, thereby improving the utilization rate of storage space.

[0067] According to an embodiment of this application, the data processing method further includes: inputting pattern data that meets a preset number of hits into a target pattern prediction model, outputting predicted pattern data, and writing the predicted pattern data into a data type library.

[0068] Pattern data meeting a preset hit count can be input into the target pattern prediction model, which outputs predicted pattern data. This predicted pattern data can be written to a data type library to improve the probability of subsequent successful matches. Furthermore, the written predicted pattern data is dynamically updated, and the hit count of each data type in the data type library is also recorded. For pattern data types with high access counts, the corresponding models can be trained first, and the predicted pattern data output by the target pattern prediction model can be expanded into the data type library. Additionally, pattern data with high hit rates can be sorted to reduce the number of comparisons and increase the priority hit rate.

[0069] According to embodiments of this application, pattern data with a high hit rate can be processed by a target pattern prediction model to obtain predicted pattern data. By expanding the data type library, the matching probability of input data with multiple data types can be increased, invalid storage operations can be reduced, and fine classification of multiple data patterns can be achieved. By writing the predicted pattern data in advance, storage efficiency can be improved.

[0070] According to embodiments of this application, the target pattern prediction model includes at least one of the following: a first target pattern prediction model for numerical prediction, a second target pattern prediction model for multimedia prediction, a third target pattern prediction model for sampling prediction, and a fourth target pattern prediction model for log prediction; wherein, the first target pattern prediction model is trained using periodic numerical fingerprint data and target pattern labels corresponding to the periodic numerical data, the periodic numerical fingerprint data being obtained by processing periodic numerical data based on an initial first target pattern prediction model; the second target pattern prediction model is trained using video frame fingerprint data and video frame pattern labels corresponding to the video frame data, the video frame fingerprint data being obtained by processing video frame data based on an initial second target pattern prediction model; the third target pattern prediction model is trained using sampled fingerprint data and sampled pattern labels corresponding to the sampled data, the sampled fingerprint data being obtained by processing sampled data based on an initial third target pattern prediction model; and the fourth target pattern prediction model is trained using template log fingerprint data and template log pattern labels corresponding to the template log data, the template log fingerprint data being obtained by processing template log data based on an initial fourth target pattern prediction model.

[0071] The target pattern prediction model can include at least one of the following: a first target pattern prediction model for numerical prediction, a second target pattern prediction model for multimedia prediction, a third target pattern prediction model for sampling prediction, and a fourth target pattern prediction model for log-based prediction. The target pattern prediction model needs to select an appropriate pattern prediction model based on the characteristics of the specific data, such as a clustering model, classification model, regression model, support perceptron, or neural network. By analyzing patterns in the data, it discovers hidden information and rules within the data and stores them as pattern data to expand the matching range of subsequent inputs with various data types.

[0072] By processing periodic numerical data using the initial first target model prediction model, periodic numerical fingerprint data can be obtained. Using the periodic numerical fingerprint data and the corresponding target pattern labels, a first target pattern prediction model can be obtained. The target pattern labels corresponding to the periodic numerical data can include various periodically changing values. For example, if the input data is 52315231 / 123456 / 52315231 / 123456…, then the target patterns can include “52315231”, “123456”, and “5231”.

[0073] By processing video frame data using the initial second-objective model prediction model, video frame fingerprint data can be obtained. For video frame pattern labels, more detailed classifications can be made based on data types, such as sports, movies, and news. For movie videos, this could include intro frames containing distributor information, outro frames, all-black frames, all-white frames, and advertising frames. Common video frames contained across multiple movies can be identified as video frame pattern labels. Using the video frame fingerprint data and the corresponding video frame pattern labels, the second-objective model prediction model can be obtained.

[0074] By processing the sampled data using the initial third-objective model prediction model, sampled fingerprint data can be obtained. Using the sampled fingerprint data and the corresponding sampled pattern labels, a third-objective pattern prediction model can be derived. For sampled data types, common features can be categorized according to time or spatial dimensions, and these common features can be defined as sampled pattern labels.

[0075] The fourth objective pattern prediction model is trained using template log fingerprint data and template log pattern labels corresponding to the template log data. The template log fingerprint data is obtained by processing the template log data based on the initial fourth objective model prediction model.

[0076] The second objective mode prediction model for multimedia prediction can be the V-net model. The first objective mode prediction model for numerical prediction, the third objective mode prediction model for sampling prediction, and the fourth objective mode prediction model for log prediction can all use the multilayer perceptron model architecture.

[0077] Figure 3 An architecture diagram of a target pattern prediction model according to an embodiment of this application is shown.

[0078] like Figure 3 As shown, the target pattern prediction model includes an input layer 310, a first hidden layer 320, a second hidden layer 330, and an output layer 340. Pattern data with a hit count greater than a preset hit count is input into the input layer 310, passes through the first hidden layer 320 and the second hidden layer 330, and is finally output by the output layer 340 as predicted pattern data. This network structure is simple and can learn the non-linear relationship between input and output. When the system load is less than a preset load threshold, missed data can be extracted and used for training to add new pattern types. It should be noted that each data type corresponds to a different target pattern prediction model.

[0079] According to embodiments of this application, for each data type, a corresponding target pattern prediction model can be trained. By processing the input data using the target pattern prediction model, predicted pattern data can be obtained. By expanding the matching data types, storage efficiency and computing resource utilization can be improved.

[0080] According to an embodiment of this application, the data processing method further includes: in response to receiving a read request, determining pattern data corresponding to the read request based on a target mapping relationship; and determining the data to be read based on the pattern data.

[0081] In response to a read request, after querying the target mapping relationship corresponding to the target pattern data key, the target address information (value) can be determined based on the target mapping relationship. Based on the determined target address information, the pattern data can be retrieved. The data to be read is further determined based on the pattern data and non-pattern data.

[0082] According to the embodiments of this application, the data segment can be determined as pattern data through the target mapping relationship. Target address information corresponding to multiple pattern data can be obtained, saving disk input / output resources. In addition, the corresponding pattern data can be obtained based on the target address information. By combining multiple pattern data, the data to be read can be obtained. Periodic or constant patterns are virtualized, and the physical space can store non-pattern data and metadata, reducing storage costs.

[0083] Figure 4 An architecture diagram of a data processing method according to an embodiment of this application is shown.

[0084] like Figure 4 As shown, input data 410 can be matched with various data types 430 contained in the data type library 420 to obtain the target data type. These data types can include numerical, multimedia, sampling, and log data. NEW data can represent non-pattern data, which can be directly stored on disk. Based on the heat statistics module 440, the hit count of a data type can be determined. Based on the training module in the backend module 470, the target pattern prediction model corresponding to the data type with the highest hit count can be trained first. Based on the prediction module, pattern data that meets the preset hit count is input into the target pattern prediction model, outputting predicted pattern data, and written to the data type library 420.

[0085] The input data 410 is matched with various data types in the data type library 420 to obtain various target patterns corresponding to each input data. The data type library 420 stores various data types. Based on the target patterns, the input data 410 is processed using a pressure balancing module. Specifically, this may include: splitting the input data 410 into multiple target pattern data, and using multi-CPU processing to evenly distribute the multiple target patterns to different threads to solve the problem of load imbalance. Finally, each thread processes the target pattern data to obtain the target address information corresponding to the target pattern data. The target mapping relationship between the various target patterns of the input data 410 and the corresponding target address information is stored in the data type library 420.

[0086] Figure 5 A flowchart of a data processing method according to another embodiment of this application is shown.

[0087] like Figure 5 As shown, the data processing method of this embodiment includes operations S510 to S580.

[0088] In operation S510, the input data is matched with various data types in the data type library to obtain various target patterns corresponding to each input data. The data type library stores various data types. Then, operation S520 is executed.

[0089] When operating S520, determine the type of data.

[0090] In operation S530, if the target data type is numeric, the input data is compared with various periodic numeric patterns contained in the numeric type to determine the target pattern of the input data, and operation S570 is executed.

[0091] In operation S540, if the target data type is multimedia, the input data is compared with the various video frame modes included in the multimedia type to determine the target mode of the input data, and operation S570 is executed.

[0092] In operation S550, if the target data type is sampling type, the input data is compared with the various sampling modes included in the sampling type to determine the target mode of the input data, and operation S570 is executed.

[0093] In operation S560, if the target data type is log type, the input data is compared with the template log pattern contained in the log type to determine the target pattern of the input data, and operation S570 is executed.

[0094] In operation S570, the input data is divided into multiple target pattern data, and operation S580 is executed.

[0095] When operating the S580, multiple threads are used to process multiple target mode data to obtain the target address information corresponding to the target mode.

[0096] Based on the above data processing method, this application also provides a data processing apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0097] Figure 6 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown.

[0098] like Figure 6 As shown, the data processing device 600 of this embodiment includes a matching module 610, a sharding polling module 620, and a storage module 630.

[0099] The matching module 610 is used to match input data with various data types in the data type library to obtain various target patterns corresponding to each input data. The data type library stores various data types.

[0100] The sharded polling module 620 is used to perform sharded polling processing on the input data based on the target pattern to obtain the target address information corresponding to the target pattern. The sharded polling processing includes splitting the input data into multiple target pattern data and using multiple threads to process multiple target pattern data.

[0101] The storage module 630 is used to store the target mapping relationship between multiple target modes of input data and corresponding target address information into the data type library.

[0102] According to embodiments of this application, by matching input data with multiple data types in a data type library, a target pattern can be determined, expanding from traditional 0 / 1 binary data to multiple data types. This achieves refined classification of multiple pattern types and improves data storage efficiency. Based on the target pattern, input data can be segmented into multiple target pattern data. Multiple threads can be used to process multiple target pattern data, dynamically balancing the computational load. This solves the problem of single-core CPU overload caused by continuous input of a single data stream, making CPU utilization more balanced. Finally, the target mapping relationship between the multiple target patterns of the input data and the target address information corresponding to the target pattern is stored in the data type library to improve access efficiency during subsequent data access.

[0103] The matching module 610 includes a fingerprint matching unit and a comparison unit.

[0104] The fingerprint matching unit is used to match the input data with fingerprints of various data types to obtain the target data type.

[0105] The comparison unit is used to compare the input data with multiple patterns included in the target data type to determine the target pattern of the input data.

[0106] The comparison unit includes: a first comparison subunit, a second comparison subunit, a third comparison subunit, and a fourth comparison subunit.

[0107] The first comparison subunit is used to compare the input data with multiple periodic numerical patterns contained in the numerical type when the target data type is numeric, and to determine the target pattern of the input data.

[0108] The second comparison subunit is used to compare the input data with the various video frame modes included in the multimedia type when the target data type is multimedia type, and to determine the target mode of the input data. The video frame modes include all-black frame mode and all-white frame mode.

[0109] The third comparison subunit is used to compare the input data with multiple sampling modes included in the sampling type when the target data type is sampling type, and to determine the target mode of the input data. The sampling modes include constant sampling mode and periodic sampling mode.

[0110] The fourth comparison subunit is used to compare the input data with the template log pattern contained in the log type when the target data type is log type, and to determine the target pattern of the input data.

[0111] The data processing apparatus 600 in this embodiment further includes a writing module.

[0112] The writing module is used to input pattern data that meets the preset number of hits into the target pattern prediction model, output the predicted pattern data, and write the predicted pattern data into the data type library.

[0113] The writing module includes: a target pattern prediction model comprising at least one of the following: a first target pattern prediction model for numerical prediction, a second target pattern prediction model for multimedia prediction, a third target pattern prediction model for sampling prediction, and a fourth target pattern prediction model for log prediction. The first target pattern prediction model is trained using periodic numerical fingerprint data and target pattern labels corresponding to the periodic numerical data, the periodic numerical fingerprint data being obtained by processing periodic numerical data based on the initial first target pattern prediction model. The second target pattern prediction model is trained using video frame fingerprint data and video frame pattern labels corresponding to the video frame data, the video frame fingerprint data being obtained by processing video frame data based on the initial second target pattern prediction model. The third target pattern prediction model is trained using sampled fingerprint data and sampled pattern labels corresponding to the sampled data, the sampled fingerprint data being obtained by processing sampled data based on the initial third target pattern prediction model. The fourth target pattern prediction model is trained using template log fingerprint data and template log pattern labels corresponding to the template log data, the template log fingerprint data being obtained by processing template log data based on the initial fourth target pattern prediction model.

[0114] The data processing apparatus 600 in this embodiment further includes a read request module and a data to be read determination module.

[0115] The read request module is used to respond to a received read request and determine the pattern data corresponding to the read request based on the target mapping relationship.

[0116] The data to be read determination module is used to determine the data to be read based on the pattern data.

[0117] According to embodiments of this application, any plurality of modules among the matching module 610, the fragmented polling module 620, and the storage module 630 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the matching module 610, the fragmented polling module 620, and the storage module 630 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the matching module 610, the fragmented polling module 620, and the storage module 630 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0118] Figure 7 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of this application is shown.

[0119] like Figure 7 As shown, an electronic device 700 according to an embodiment of this application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0120] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0121] According to embodiments of this application, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0122] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the data processing method according to the embodiments of this application.

[0123] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0124] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data processing methods provided in the embodiments of this application.

[0125] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0126] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0127] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0128] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A data processing method, characterized in that, include: The input data is matched with various data types in the data type library to obtain various target patterns corresponding to each input data. The data type library stores various data types. Based on the target pattern, the input data is processed by sharding and polling to obtain target address information corresponding to the target pattern. The sharding and polling process includes splitting the input data into multiple target pattern data and using multiple threads to process multiple target pattern data. The target mapping relationship between the various target modes of the input data and the corresponding target address information is stored in the data type library.

2. The method according to claim 1, characterized in that, The process of matching input data with various data types in a data type library to obtain multiple target patterns corresponding to each input data includes: The input data is matched with fingerprints of multiple data types to obtain the target data type; The input data is compared with multiple patterns included in the target data type to determine the target pattern of the input data.

3. The method according to claim 2, characterized in that, The step of comparing the input data with multiple patterns included in the target data type to determine the target pattern of the input data includes: When the target data type is numeric, the input data is compared with multiple periodic numerical patterns contained in the numeric type to determine the target pattern of the input data; When the target data type is multimedia, the input data is compared with the various video frame modes included in the multimedia type to determine the target mode of the input data. The video frame modes include all-black frame mode and all-white frame mode. When the target data type is a sampling type, the input data is compared with multiple sampling modes included in the sampling type to determine the target mode of the input data. The sampling modes include constant sampling mode and periodic sampling mode. If the target data type is log type, the input data is compared with the template log pattern contained in the log type to determine the target pattern of the input data.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The pattern data that meets the preset number of hits is input into the target pattern prediction model, the predicted pattern data is output, and the predicted pattern data is written into the data type library.

5. The method according to claim 4, characterized in that, The target pattern prediction model includes at least one of the following: The four target model prediction models are: a first target model prediction model for numerical forecasting, a second target model prediction model for multimedia forecasting, a third target model prediction model for sampling forecasting, and a fourth target model prediction model for log-based forecasting; among them, The first target pattern prediction model is trained using periodic numerical fingerprint data and target pattern labels corresponding to the periodic numerical data. The periodic numerical fingerprint data is obtained by processing the periodic numerical data based on the initial first target model prediction model. The second target pattern prediction model is trained using video frame fingerprint data and video frame pattern labels corresponding to the video frame data. The video frame fingerprint data is obtained by processing the video frame data based on the initial second target model prediction model. The third target pattern prediction model is trained using sampled fingerprint data and sampled pattern labels corresponding to the sampled data. The sampled fingerprint data is obtained by processing the sampled data based on the initial third target model prediction model. The fourth target pattern prediction model is trained using template log fingerprint data and template log pattern labels corresponding to the template log data. The template log fingerprint data is obtained by processing the template log data based on the initial fourth target model prediction model.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: In response to receiving a read request, the pattern data corresponding to the read request is determined according to the target mapping relationship; Based on the pattern data, determine the data to be read.

7. A data processing apparatus, characterized in that, The device includes: The matching module is used to match input data with various data types in the data type library to obtain various target patterns corresponding to each input data. The data type library stores various data types. The sharding polling module is used to perform sharding polling processing on the input data based on the target mode to obtain target address information corresponding to the target mode. The sharding polling processing includes splitting the input data into multiple target mode data and using multiple threads to process multiple target mode data. The storage module is used to store the target mapping relationship between the various target modes of the input data and the corresponding target address information into the data type library.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Images