Data processing method and related equipment

Abstract

The embodiment of the invention discloses a data processing method, which can be used for preprocessing stream data to obtain a first tensor, and also can be used for preprocessing sequence data to obtain the first tensor after converting batch processing data into the sequence data. The first tensor comprises an object dimension, a feature dimension and a time dimension, so that information of stream data and batch processing data can be flexibly stored, and the first tensor supports data processing in a real-time scene and also supports data processing in a batch processing scene. Therefore, the batch processing data and the stream data can be preprocessed into the form of the first tensor, and then the corresponding processing result can be obtained according to the first tensor. Therefore, the data processing method not only can process the streaming data, but also can process the batch processing data, so that a user does not need to develop the data processing flow for the batch processing data and the streaming data respectively, and the development process is simplified.

Description

[0001] This application claims the priority of an application filed with the National Intellectual Property Administration on July 21, 2023, with Chinese application number 202310905679.8 and invention name “A data processing method, device and system”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method and related equipment. Background Art

[0003] In application fields such as healthcare, autonomous driving, finance, and visual recognition, data processing is often achieved with the help of computing frameworks such as TensorFlow and PyTorch. For example, AI models can be trained and deployed through computing frameworks, thereby achieving data processing and application through AI models.

[0004] In actual application scenarios, the data to be processed may be batch data that needs to be processed in batches, or stream data that needs to be processed in real time. However, existing computing frameworks are usually suitable for processing batch data, but have weak processing capabilities for stream data. When it is necessary to process stream data, developers usually need to use real-time computing frameworks such as Apache Flink to implement the processing of stream data.

[0005] It can be seen that in scenarios with both batch data and streaming data, relevant personnel in actual applications need to develop data processing processes for batch data and streaming data respectively. In other words, two sets of data processing processes need to be developed, and the complexity and cost of the development process are relatively high. Summary of the invention

[0006] The embodiment of the present application provides a data processing method, which can solve the problem that relevant personnel in practical applications need to develop data processing processes for batch data and stream data respectively, resulting in the complexity and high cost of the development process. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.

[0007] A first aspect of the present application provides a data processing method, which is applied to a data processing platform. The data processing platform includes a first processing module and a second processing module, wherein the first processing module provides a configuration interface, the configuration interface is used to instruct a user to input configuration information, the first processing module is implemented based on a first programming language, the configuration information is described based on the first programming language, and the second processing module is implemented based on a second programming language, which is different from the first programming language.

[0008] The method includes: a first processing module or a second processing module acquires input data, the input data is stream data, or is sequence data obtained based on batch data; the second processing module preprocesses the input data to obtain a first tensor, the first tensor includes three dimensions, the three dimensions are an object dimension corresponding to an object in the input data, a feature dimension corresponding to a feature of the object, and a time dimension; the first processing module or the second processing module obtains a processing result corresponding to the input data based on the first tensor.

[0009] In the first aspect, the stream data can be preprocessed to obtain a first tensor, or the batch data can be converted into sequence data and then the sequence data can be preprocessed to obtain the first tensor. The first tensor includes an object dimension, a feature dimension, and a time dimension, so that the information of the stream data and the batch data can be flexibly stored, so that the first tensor supports both data processing in real-time scenarios and data processing in batch scenarios. In this way, both batch data and stream data can be preprocessed into the form of a first tensor, and then, can be processed according to the first tensor and the configuration information input by the user to obtain the corresponding processing results.

[0010] It can be seen that in the first aspect, the data processing method can process both streaming data and batch data, that is, in the first aspect, a batch-stream integrated data processing method is implemented, so that users in practical applications do not need to develop data processing flows for batch data and streaming data separately, simplifying the development process. In addition, since streaming data and batch data can be converted into the form of a first tensor for subsequent data processing operations, the inconsistency caused by the difference in the processing methods of streaming data and batch data in traditional technologies in subsequent feature extraction, processing and other stages can be avoided, providing a better data foundation for subsequent data processing operations.

[0011] Furthermore, in the first aspect, a configuration interface for implementing user interaction may be provided by using a first processing module of a first programming language, and the user may input corresponding configuration information through one or more configuration items provided on the configuration interface.

[0012] The first programming language can be a programming language that conforms to the user's usage habits (such as Python, etc.). In this way, no matter in the scenario of batch data processing or stream data processing, the user can perform efficient data configuration and corresponding data processing in a familiar development environment, so that the user can use a unified and easy-to-understand method to carry out batch and stream integrated data development, and realize efficient management and execution of various data processing tasks.

[0013] The second programming language used by the second processing module is different from the first programming language. The second programming language may be a programming language with more powerful performance.

[0014] For example, in some examples, the first programming language may be Python, and the second programming language may be C++. In this example, the data processing platform cleverly combines the ease of use of Python and the efficiency of C++ to achieve batch and stream integrated data processing, which is not only efficient but also easy to use.

[0015] The second processing module can be responsible for efficient underlying data processing and data processing operations such as real-time computing tasks. The second processing module may not be perceived by the user, that is, the user can interact with the computing device cluster through the first processing module, input configuration information, and obtain corresponding processing results, but the user does not need to perceive the specific processing process of the second processing module. It can be seen that in this example, the data processing platform can be an end-to-end batch-stream integrated data processing platform with strong ease of use and practicality.

[0016] In a possible implementation of the first aspect, the method also includes: the first processing module receives configuration information, the configuration information includes processing logic; the first processing module transmits the processing logic to the second processing module; the first processing module or the second processing module obtains the processing result corresponding to the input data according to the first tensor, including: the second processing module triggers the first processing module or the second processing module to process the first tensor according to the processing logic to obtain the processing result corresponding to the input data.

[0017] In this possible implementation, the user can configure the data processing method of the input data through the configuration interface of the first processing module, that is, configure the processing logic. Then, the first processing module transmits the configuration information to the second processing module, so that the second processing module is responsible for efficient underlying data processing and data processing operations such as real-time computing tasks. The second processing module may not be perceived by the user, that is, the user can interact with the data processing platform through the first processing module and input the processing logic, but the user does not need to perceive the specific processing process of the second processing module.

[0018] In a possible implementation of the first aspect, the second processing module triggers the first processing module or the second processing module to obtain a processing result corresponding to the input data according to the first tensor and the processing logic, including: the second processing module writes the first tensor to the storage space of the shared memory, and the storage space is used to store tensors including three dimensions; the second processing module triggers the first processing module or the second processing module to read the first tensor from the storage space to obtain the processing result corresponding to the input data according to the read first tensor and the processing logic.

[0019] In traditional technology, real-time processing frameworks such as Apache Flink are oriented to Java developers, while computing frameworks such as TensorFlow and PyTorch are implemented through Python. In traditional technology, although the real-time processing framework provides a Python application programming interface (API), the performance of the API is poor, which may cause Python developers to face performance bottlenecks when using Apache Flink, and the efficiency of information interaction is usually low. In addition, in traditional technology, when information is exchanged between modules of different programming languages, it is usually necessary to convert the data to overcome the obstacles brought by different programming languages, and then store the converted data in a space such as a hard disk. Then, one of the modules needs to read the converted data from the space through more complex interaction instructions such as access instructions and data reading instructions before data parsing and processing can be performed. This traditional information interaction method is inefficient and requires multiple information interactions and data copies.

[0020] In this possible implementation, shared memory, an efficient middleware, can be used to efficiently transmit information between the first processing module and the second processing module based on a three-dimensional storage format.

[0021] Specifically, a storage space may be provided in the shared memory to store a tensor including three dimensions through the storage space.

[0022] After obtaining the first tensor, the second processing module can write the first tensor into the storage space of the shared memory. By reading data through shared memory, multiple copies of data can be avoided and the processing efficiency of data can be improved. In addition, the first tensor is stored in the shared memory in the form of a tensor including three dimensions, so that the first processing module or the second processing module can quickly and easily identify and find the required data from the shared memory based on the information of each dimension, overcoming the information interaction barriers caused by different programming languages. It can be seen that by sharing memory, the data interaction between the first processing module and the second processing module about the first tensor can be realized, which can greatly improve the interaction performance between the first processing module and the second processing module, and overcome the information interaction barriers caused by different programming languages.

[0023] In a possible implementation of the first aspect, the configuration information of the first tensor includes a trigger condition; the first processing module or the second processing module obtains the processing result corresponding to the input data based on the first tensor, including: when it is determined based on the first tensor that the trigger condition is met, the second processing module obtains the processing result corresponding to the input data based on the first tensor.

[0024] In this possible implementation, the trigger condition is used to indicate that the first tensor and other data required for subsequent data processing are ready, thereby indicating that the subsequent computing engine can process according to the first tensor and processing logic, etc., to obtain the processing result corresponding to the input data.

[0025] In a possible implementation of the first aspect, the configuration information of the first tensor includes preprocessing configuration information, and the preprocessing configuration information is used to configure one or more dimensions of the first tensor; the second processing module preprocesses the input data to obtain the first tensor, including: the second processing module preprocesses the input data according to the preprocessing configuration information to obtain the first tensor.

[0026] In this possible implementation, the user can configure the processing method for the input data through preprocessing configuration information, so that the generated first tensor can meet the needs of subsequent calculations and processing, reducing the processing steps and processing difficulty of the first tensor in the subsequent processing process.

[0027] In a possible implementation of the first aspect, the storage space includes a first storage subspace and a second storage subspace; the second processing module writes the first tensor into the storage space of the shared memory, including: the second processing module stores the first tensor in the first storage subspace based on the time information of the first tensor; when the first storage subspace is full of tensors within a specified time period in the time dimension, the second processing module stores the tensors within the specified time period in the second storage subspace, so that the first processing module and / or the second processing module reads the tensors within the specified time period from the second storage subspace.

[0028] In this possible implementation, in order to facilitate time series calculations and save memory at the same time, the storage location can be adjusted in the shared memory based on the writing status of the tensor, so that users can easily obtain the required time series window data and perform efficient calculations in the time series calculations, greatly reducing the latency of the time series calculations.

[0029] In a possible implementation of the first aspect, a second tensor is also stored in the shared memory, and the second tensor includes two dimensions, namely, an object dimension and a feature dimension. In the second tensor, the feature value of each object is the latest feature value corresponding to the corresponding object, and the latest feature value is updated by the second processing module according to the input data.

[0030] In this possible implementation, it is convenient to quickly obtain the latest state value of the features of each object through the second tensor in scenarios such as incremental calculation or cross-sectional calculation, without the need for more complex and time-consuming data retrieval operations in shared memory, thereby greatly improving the computational efficiency of scenarios such as incremental calculation and cross-sectional calculation.

[0031] In a possible implementation of the first aspect, a data processing platform is used to train an artificial intelligence AI model; the second processing module triggers the first processing module or the second processing module based on a trigger condition to obtain a processing result corresponding to the input data according to the first tensor and the processing logic, including: at the nth moment, the second processing module triggers the first processing module to obtain the nth group of training data based on the content in the storage space at the nth moment in the shared memory and / or the second tensor at the nth moment, so that the first processing module performs the nth batch of training on the AI ​​model according to the nth group of training data, where n is a positive integer; during the training process of the nth batch, at the n+1th moment, the second processing module triggers the first processing module to obtain the n+1th group of training data based on the content in the storage space at the n+1th moment in the shared memory and / or the second tensor at the n+1th moment, so that after the training of the nth batch is completed, the first processing module performs the n+1th batch of training on the AI ​​model according to the n+1th group of training data.

[0032] In the traditional AI model training process, usually all batch data can only be directly obtained at one time as training data, and training is performed based on all the batch data.

[0033] In this possible implementation, considering that in a real-time scenario, the content of the storage space in the shared memory and the data of the second tensor can be updated with the real-time data, so that the training data can be obtained from the storage space and / or the second tensor at different time nodes, and pipelined, multiple batches of training can be performed. In this pipeline training process, the data preparation time is less than the time required for the first processing module to perform a batch of training. In this way, the data preparation time can be covered by the training process of the previous stage without adding additional processing time, so that AI model training can be carried out continuously and uninterruptedly without waiting for data, thereby achieving maximum throughput and improving training efficiency.

[0034] In a possible implementation manner of the first aspect, the data processing method is applied to the financial field, and the input data includes financial data.

[0035] The second aspect of the present application provides a data processing device, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as a processing module.

[0036] A third aspect of the present application provides a computing device, which includes at least one processor, a memory, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0037] In a fourth aspect, the present application provides a computing device cluster, which includes at least one computing device, and the at least one computing device includes at least one processor, a memory, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0038] The fifth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0039] In a sixth aspect, the present application provides a computer program product storing one or more computer-executable instructions. The computer program product includes computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0040] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting a computing device or a computing device cluster to implement the functions involved in the first aspect or any possible implementation of the first aspect. In a possible design, the chip system may also include a memory, which is used to store necessary program instructions and data. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0041] Among them, the technical effects brought about by the second to seventh aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the architecture of a data center provided in an embodiment of the present application;

[0043] Figure 2a is a schematic diagram of an exemplary system provided in an embodiment of the present application;

[0044] Figure 2b is another exemplary system schematic diagram provided in an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of an embodiment of a data processing method provided in an embodiment of the present application;

[0046] Figure 4 is an exemplary schematic diagram of a data processing platform provided in an embodiment of the present application;

[0047] Figure 5a This is a schematic diagram of a data processing flow for a real-time scenario provided by an embodiment of the present application;

[0048] Figure 5b This is a schematic diagram of a data processing flow for an offline scenario provided in an embodiment of the present application;

[0049] Figure 6 is an exemplary schematic diagram of a storage space provided in an embodiment of the present application;

[0050] Figure 7 is an exemplary schematic diagram of a second tensor provided in an embodiment of the present application;

[0051] Figure 8 It is an exemplary schematic diagram of pipeline training of an AI model provided in an embodiment of the present application;

[0052] Figure 9a is an exemplary flow chart of a data preparation stage provided in an embodiment of the present application;

[0053] Figure 9b is an exemplary flowchart of a training process of the nth batch provided in an embodiment of the present application;

[0054] Fig.10 is a schematic diagram of an exemplary distributed parallel mode provided in an embodiment of the present application;

[0055] Fig.11 This is a schematic diagram of an exemplary process of performing reasoning based on input data in a real-time scenario provided by an embodiment of the present application;

[0056] Fig.12 is a schematic diagram of an embodiment of a data processing device provided in an embodiment of the present application;

[0057] Fig.13 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0058] Fig.14 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0059] Fig.15 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0061] Those skilled in the art may appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0062] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of singular or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, and this is merely a description of the way in which objects with the same attributes are described in the embodiments of the present application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0063] At present, AI model training and deployment are often implemented through AI frameworks such as TensorFlow and PyTorch, thereby realizing data processing and application related to AI models.

[0064] However, in scenarios where there is both batch data and streaming data, relevant personnel in actual applications need to develop data processing flows for batch data and streaming data respectively. In other words, two sets of data processing flows need to be developed, and the development process is complex and costly.

[0065] In addition, when real-time data and batch data are processed through different data processing processes, data consistency issues may arise. Since real-time data and batch data may be processed differently, this may lead to inconsistent results in stages such as feature extraction. This inconsistency will affect the subsequent training and prediction effects of AI models.

[0066] The embodiment of the present application provides a data processing method, which can solve the problem that relevant personnel in actual applications need to develop data processing processes for batch data and stream data respectively, which leads to the complexity and high cost of the development process, and can overcome the data consistency problem that may arise when processing real-time data and batch data through different data processing processes.

[0067] The method of the embodiment of the present application may be applied to a computing device cluster, which may include one or more computing devices.

[0068] The type of any computing device is not limited herein. Exemplarily, any computing device may be a terminal device, or a server, a container, or a virtual machine.

[0069] The computing device cluster can provide a data processing platform to implement the data processing method of the embodiment of the present application through the data processing platform.

[0070] The specific forms of the computing device cluster and the corresponding data processing platform are not limited here.

[0071] In one example, the computing device cluster can be used to implement a cloud management platform. In other words, the data processing platform of the embodiment of the present application can be implemented through a cloud management platform.

[0072] The cloud management platform is used to manage the infrastructure for providing cloud services, and can provide computing, network and storage capabilities based on services of hardware resources and software resources. Exemplarily, the cloud management platform can include one or more data centers to provide cloud resources through one or more data centers.

[0073] Combine the following Figure 1 An architectural diagram is shown to introduce a data center.

[0074] Figure 1 In the data center, the cloud management platform communicates with one or more servers (such as Figure 1The server 1 and server 2 in the server exchange information. The server includes a hardware layer and a software layer. The hardware layer includes the hardware configured by the server, wherein the PCI device may be, for example, a network card, a graphics processing unit (GPU), an offload card, etc., which can be inserted into the peripheral component interconnect standard (PCI) or peripheral component interconnect express (PCIe) slot of the server. The software layer includes an operating system installed and running on the server (relative to the operating system of the virtual machine, it can be called the host operating system). The host operating system is provided with a virtual machine manager (also called a hypervisor). The role of the virtual machine manager is to realize the computing virtualization, network virtualization and storage virtualization of the virtual machine, and is responsible for managing the virtual machine. Among them, a virtual machine (virtual machine) refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. Figure 1 In the system architecture shown, a data center is equipped with multiple servers, which can be used to run virtual machines. The specifications of each virtual machine can be the same or different. Among them, the virtual machine can also be called a cloud server (elastic compute service, ECS), elastic instance, etc. Different cloud service providers may have different names.

[0075] In one example of an embodiment of the present application, the cloud management platform may be a public cloud platform. In this case, cloud service providers such as individuals or software developers with cloud resource development capabilities may provide cloud services to users. Users obtain cloud services through the Internet but do not own cloud computing resources.

[0076] Specifically, in Figure 1 In the example shown, the cloud management platform can provide an access interface (such as an interface or application programming interface (API)), and users of the cloud management platform and cloud service providers can operate the client remote access access interface to register a cloud account and password on the cloud management platform, and log in to the cloud management platform after the cloud management platform successfully authenticates the cloud account and password, thereby creating, managing, logging in and operating virtual machines in the cloud data center, and performing corresponding tasks through the cloud resources of the cloud management platform.

[0077] For example, if Figure 2a In the example shown, the data processing platform that implements the data processing method in the embodiment of the present application can be provided to users in the form of cloud services.

[0078] When it is necessary to perform data processing tasks on stream data or batch data, some enterprises, organizations or individuals can purchase cloud services that include the data processing platform, thereby performing related data processing tasks through the cloud resources of the cloud management platform and obtaining the processing results of the input data from the cloud management platform.

[0079] The data processing platform can be implemented based on cloud resources such as one or more cloud data centers in the cloud management platform.

[0080] The data processing platform can be provided to users as an independent cloud service, and can also be compatible with computing frameworks such as TensorFlow and PyTorch. That is to say, it can be used as an operator in computing frameworks such as TensorFlow and PyTorch, and provide preprocessing, calculation, feature extraction and / or other data processing functions for real-time data and offline data in the computing framework.

[0081] Of course, the cloud management platform may also be other types of cloud management platforms, which is not limited in the embodiments of the present application.

[0082] In another example, Figure 2b As shown, the data processing platform can be provided to the user in the form of a software product or a functional service in the software product. The software product can be deployed in the user's terminal device or the server cluster of the enterprise user. The user can purchase a software license to obtain the right to use the software product, and can implement the data processing method of the embodiment of the present application through the software product within the license period.

[0083] Taking the financial field as an example, the actual application of the data processing platform that implements the data processing method is exemplified below.

[0084] In an exemplary application scenario, the user may be a data analyst for a financial enterprise, who uses AI models to process and analyze real-time data in the stock market through computing frameworks such as TensorFlow and PyTorch. The computing framework can be deployed in a client device or provided through cloud resources of a cloud management platform. However, the current computing framework has weak processing capabilities for real-time data. Therefore, in this example, the cloud management platform can provide users with services that support both batch processing of batch data and real-time processing of streaming data, thereby improving the processing performance of the computing framework and reducing the user's development costs.

[0085] It can be understood that the above example is only an illustrative introduction to an application scenario, and the data processing method of the embodiment of the present application can be applied to multiple fields and is not limited here.

[0086] In the embodiment of the present application, when the data processing method in the embodiment of the present application is implemented by the data processing platform, both the processing of streaming data in real-time scenarios and the processing of batch data in offline scenarios can be implemented. That is to say, in the embodiment of the present application, the data processing platform supports both the processing of batch data and the real-time processing of streaming data, without the need to develop two sets of data processing systems for batch data and streaming data respectively.

[0087] The data processing platform is introduced below.

[0088] The data processing platform includes a first processing module and a second processing module, wherein the first processing module provides a configuration interface, the configuration interface is used to instruct a user to input configuration information, the first processing module is implemented based on a first programming language, the configuration information is described based on the first programming language, and the second processing module is implemented based on a second programming language, which is different from the first programming language.

[0089] In the embodiment of the present application, a configuration interface for realizing user interaction is provided by the first processing module, and the user can input corresponding configuration information through one or more configuration items provided on the configuration interface. For example, the user can input configuration information to the computing device cluster through the interaction between an electronic device such as a device and the computing device cluster; or, when the computing device cluster is a terminal device, the user can also input the configuration information to the computing device cluster by touch, or by input instructions generated by components such as a keyboard and a mouse.

[0090] The first programming language may be a programming language that conforms to the user's usage habits. In this way, no matter in the scenario of processing batch data or processing stream data, the user can perform efficient data configuration and corresponding data processing in a familiar development environment, so that the user can use a unified and easy-to-understand method to perform batch and stream integrated data development, thereby achieving efficient management and execution of various data processing tasks.

[0091] For example, in some examples, Python is the preferred language for many data analysts because of its simplicity, ease of use and rich scientific computing libraries. Data analysts in fields such as finance, Internet of Things, e-commerce platforms, and smart manufacturing are accustomed to using computing frameworks such as TensorFlow and PyTorch based on the Python programming language to perform data analysis.

[0092] To this end, in this example, the first programming language may be the Python programming language. In this way, the first processing module may be implemented based on the Python programming language, and the user may input configuration information based on the Python programming language through the configuration interface provided by the first processing module. This method of inputting configuration information is a method that the user is currently accustomed to and good at, so that the user does not need to change and learn a new programming method, which greatly reduces the user's learning cost, improves configuration efficiency, thereby improving the overall data processing efficiency, and providing a good user experience.

[0093] Of course, in some other examples, with the development of technology or based on actual application scenarios, the first programming language may also be a programming language other than Python.

[0094] The specific content of the configuration information input by the user through the configuration interface is not limited here. The specific content and specific application of the configuration information are described in the subsequent configuration information related embodiments (for example Figure 4 as well as Figure 5a and Figure 5b The relevant embodiments are exemplarily introduced in detail and will not be described in detail here.

[0095] The data processing platform also includes a second processing module, which is implemented based on a second programming language.

[0096] The second programming language used by the second processing module is different from the first programming language. The second programming language can be a programming language with more powerful performance.

[0097] For example, in some examples, considering the efficiency and high performance of C++, the second programming language may be C++. In this example, since the user does not need to perceive the specific operation of the second processing module, there is no need to learn C++ development skills.

[0098] Moreover, in this example, the first programming language may be python. In this example, the data processing platform skillfully combines the ease of use of python and the efficiency of C++, and realizes batch-stream integrated data processing, which is not only efficient but also easy to use.

[0099] The second processing module can be responsible for efficient underlying data processing and data processing operations such as real-time computing tasks. The second processing module may not be perceived by the user, that is, the user can interact with the computing device cluster through the first processing module, input configuration information, and obtain corresponding processing results, but the user does not need to perceive the specific processing process of the second processing module, so the user's learning cost will not be increased. It can be seen that in this example, the data processing platform can be an end-to-end batch-stream integrated data processing platform with strong ease of use and practicality.

[0100] Based on the above data processing platform, such as Figure 3 As shown, the data processing method may include steps 301-303.

[0101] Step 301: The first processing module or the second processing module obtains input data.

[0102] The input data is stream data, or sequence data obtained based on batch data.

[0103] In the embodiment of the present application, the input data may come from a variety of sources. For example, the input data may be pre-stored in the computing device cluster executing the embodiment of the present application, or may be transmitted to the computing device cluster by other devices (such as a database or a client device), or may be collected by the computing device cluster through a web page, other applications or sensors, or may be obtained by the computing device cluster after processing image data, voice data, point cloud data, text, etc.

[0104] In some examples, the input data may be streaming data.

[0105] Stream data can be a set of data sequences that arrive in order, in large quantities, quickly, and continuously. In many scenarios, the data arriving in the data sequence has the characteristics of order, large quantities, and fast speed. In actual scenarios, some of the received stream data may be missing, delayed, or out of order. In general, stream data can be regarded as a dynamic data set that grows over time.

[0106] Each data in the stream data usually has time information. For example, each data in the stream data may be accompanied by a corresponding timestamp.

[0107] Stream data is usually obtained in real-time scenarios. The delay of stream data in real-time scenarios is small, for example, it can be a delay of milliseconds, seconds, or minutes. For example, in the financial field, real-time updated price information of multiple stocks can be obtained as stream data. For example, in this stream data, a piece of data containing the latest stock prices of 1,000 stocks is obtained every 3 seconds.

[0108] In real-time scenarios, stream data is often processed by processing one piece of stream data each time it is received, which has high latency requirements.

[0109] In other examples, the input data may also be obtained based on batch data.

[0110] For example, batch data can be processed to obtain sequence data.

[0111] Batch data refers to a set of data that can be used for batch processing. Batch processing refers to obtaining a large amount of data at one time for processing.

[0112] Batch data is usually data in offline scenarios. For example, in the financial field, all stock price changes of 1,000 stocks in the day before the current moment can be used as a set of batch data.

[0113] It can be seen that the data processing methods of stream data and batch data are quite different. At present, it is usually difficult to process stream data and batch data through the same set of data processing procedures.

[0114] In the embodiment of the present application, batch data can be processed to obtain sequence data.

[0115] The method of obtaining sequence data based on batch data is not limited here.

[0116] In one example, each data in the batch data may be arranged according to the time information (eg, timestamp) of each data in the batch data to obtain sequence data.

[0117] For example, the batch data includes the stock price changes of 100 stocks one day before the current moment.

[0118] The batch data includes multiple pieces of data, each piece of data corresponds to a timestamp in the previous day, and each piece of data contains the stock prices of 100 stocks at the moment corresponding to the timestamp.

[0119] In an example, multiple pieces of data in the batch data may be sorted according to the sequence of timestamps corresponding to each piece of data in the batch data to obtain sequence data corresponding to the batch data.

[0120] In other examples, based on relevant configurations of the user and / or preset configurations of the data processing platform, etc., the batch data is processed to obtain corresponding sequence data.

[0121] For example, in another example, the batch data includes multiple pieces of data, each piece of data includes the change in the stock price of a stock over time in the previous day. Then, the price data of each stock in the batch data can also be sorted according to the stock name, priority, average price, etc. to obtain corresponding sequence data.

[0122] It can be seen that in the embodiments of the present application, there may be multiple ways to process batch data to obtain corresponding sequence data. For example, multiple data in the batch data may be sorted based on the relevant configuration of the user, and this is not limited in the embodiments of the present application.

[0123] The sequence data may carry time information of the batch data.

[0124] The data format of the sequence data is relatively close to the stream data, both of which are a set of data arranged in sequence. Therefore, it is convenient to preprocess the sequence data or stream data in the preprocessing stage, without making it difficult to preprocess through the same preprocessing engine due to the large difference in data format.

[0125] Step 302: The second processing module preprocesses the input data to obtain a first tensor.

[0126] The first tensor includes three dimensions, which are an object dimension corresponding to the object in the input data, a feature dimension corresponding to the feature of the object, and a time dimension.

[0127] In an embodiment of the present application, the input data may be preprocessed by a preset preprocessing engine to output a first tensor.

[0128] Specifically, the input data may be input into the preprocessing engine to output a first tensor about the input data through the preprocessing engine, so that the input data may be converted into a form of a tensor including three dimensions.

[0129] The specific process of the preprocessing engine preprocessing the input data is not limited here.

[0130] Exemplarily, the preprocessing engine performs operations such as filtering, projection, and / or aggregation on the input data to convert the input data into the form of a first tensor.

[0131] The preprocessing rules may be preconfigured by the developer of the preprocessing engine, or may be configured by the user; or, a part of the preprocessing rules may be preconfigured by the developer of the preprocessing engine, and another part of the preprocessing rules may be configured by the user.

[0132] For example, in a real-time scenario, the stream data used as input data may be a stream data arriving every 2 seconds, and each stream data includes the new stock prices of 100 stocks. The user can configure the preprocessing rule to generate a first tensor corresponding to the stream data every 10 seconds according to the actual data processing requirements. Then, the preprocessing engine can receive 5 stream data within 10 seconds, parse the 5 stream data, determine the timestamp and other information corresponding to each stream data, and convert the 5 parsed stream data into the form of a first tensor including three dimensions.

[0133] The first tensor may include three dimensions, which are an object dimension corresponding to the object in the input data, a feature dimension corresponding to the feature of the object, and a time dimension.

[0134] The object dimension is used to describe the object corresponding to the input data. For example, in the financial field, the input data is the real-time updated price information of multiple stocks. Then, the object in the input data is the stock, and the object dimension of the first tensor is used to record the name, number and other information of the stock in the input data.

[0135] The feature dimension is used to describe the features corresponding to the object. The features of an object are used to describe information such as the properties of the object. For example, in the financial field, the input data is the real-time updated price information of multiple stocks. Then, the feature of the object in the input data is the price, and the feature dimension of the first tensor is used to record the price information of the corresponding stock in the input data.

[0136] The time dimension is used to describe the time information of the input data. The input data may carry time information such as a corresponding timestamp, and after preprocessing, the time information of the input data is recorded through the time dimension of the first tensor.

[0137] In the embodiment of the present application, the information of the input data in different scenarios is flexibly recorded through the data of the three dimensions. That is to say, when the input data is stream data, the time information carried by the stream data as real-time data can be recorded through the time dimension in the first tensor. In addition, the information on the object dimension and feature dimension of the stream data itself can also be recorded through the first tensor. When the input data is sequence data obtained based on batch data, the information of the batch data can also be comprehensively recorded through the three dimensions in the first tensor, providing a better data foundation for subsequent data processing.

[0138] In other words, both streaming data and batch processing data can be converted into the data form of the first tensor, so that data processing can be performed in the data form of the first tensor later, avoiding the situation where batch and stream integrated processing cannot be performed due to the difference in data form between real-time scenarios and batch processing scenarios.

[0139] Step 303: The first processing module or the second processing module obtains a processing result corresponding to the input data according to the first tensor.

[0140] The user in the embodiment of the present application may refer to a user who uses the client or other electronic devices to perform human-computer interaction with a computing device cluster, or may refer to a user account logged into the data processing platform.

[0141] In one example, the data processing method of the embodiment of the present application is applied to a software product and provided to the customer in the form of a software product or a functional service in the software product. Then, the customer can download and install the software product on the client or the customer's server cluster, and can purchase a software license to obtain the right to use the software product, and can implement the data processing method of the embodiment of the present application through the software product within the license period. At this time, the user can be the user account that installed the software product, and in some scenarios, it can also be the user account that purchased the software license.

[0142] In another example, the data processing method of the embodiment of the present application is applied to a cloud management platform and provided to customers in the form of cloud services. Then, the customer can register an account on the cloud management platform to purchase the cloud service through the account, thereby implementing the data processing method of the embodiment of the present application through the cloud service. At this time, the user can be the user account that purchased the cloud service.

[0143] In the embodiment of the present application, the first tensor can be processed by a computing engine to obtain a processing result corresponding to the input data.

[0144] Based on the requirements of different application scenarios, the computing engine can perform processing operations such as calculation, feature extraction, and data processing based on AI models on the first tensor. If the input data is streaming data, the data processing method in the real-time scenario can be used to process it based on the first tensor; if the input data is sequence data obtained based on batch data, the data processing method in the offline scenario can be used to process it based on the first tensor.

[0145] In the embodiment of the present application, there is no limitation on the data processing process of the first tensor.

[0146] In some embodiments, the first processing module may receive configuration information through a configuration interface, and the computing engine may process the first tensor according to the configuration information to obtain a processing result corresponding to the input data. The specific content of the configuration information is not limited here.

[0147] In some examples, the configuration information may include configuration information of the first tensor, such as the dimensional form of the first tensor, the timing of reading the first tensor, and the timing of calculating the first tensor.

[0148] For example, if a piece of data is obtained every 2 seconds in the streaming data, and the user needs to perform calculations on the input data within every 20 seconds, the user can configure the aggregation logic in the configuration information so that the computing device cluster aggregates the 10 pieces of data obtained within every 20 seconds to generate a first tensor, and the time dimension of the first tensor can correspond to 20 seconds. In addition, the user can also configure in the configuration information that a calculation is performed every time a first tensor is generated, then the computing device cluster can trigger the corresponding calculation after each first tensor is generated according to the configuration information.

[0149] In another example, the configuration information includes processing logic, and the first tensor is processed according to the processing logic.

[0150] The processing logic is used to indicate a data processing method for the first tensor. Exemplarily, the processing logic may specifically include configurations such as relevant algorithms, parameters, and modules for processing the first tensor.

[0151] In this way, the computing device cluster can obtain the processing result corresponding to the input data according to the first tensor and the configuration information input by the user.

[0152] In addition, in some embodiments, an efficient information interaction method between the first processing module and the second processing module in the data processing platform is also provided.

[0153] Next, combine Figure 4 The example shown provides an exemplary introduction to the specific data processing flow of the data processing method implemented by the first processing module and the second processing module based on different configuration information of the processing logic, preprocessing configuration information and trigger conditions.

[0154] 1. Configuration information includes processing logic.

[0155] In some embodiments, the method further comprises:

[0156] The first processing module receives configuration information, where the configuration information includes processing logic;

[0157] The first processing module transfers the processing logic to the second processing module;

[0158] The first processing module or the second processing module obtains a processing result corresponding to the input data according to the first tensor, including:

[0159] The second processing module triggers the first processing module or the second processing module to process the first tensor according to the processing logic to obtain a processing result corresponding to the input data.

[0160] In the embodiment of the present application, the configuration information includes processing logic. After obtaining the configuration information input by the user, the first processing module can transmit the processing logic to the second processing module to push down the processing logic, so that the second processing module triggers the first processing module or the second processing module obtains the processing result corresponding to the input data according to the first tensor and the processing logic.

[0161] For example, the processing logic may include information such as the scene corresponding to the first tensor to indicate that the input data corresponding to the first tensor is batch data and needs to be calculated offline by the first processing module. Then, after obtaining the first tensor, the second processing module may instruct the first processing module to calculate the first tensor according to the first tensor and the processing logic through the computing engine (e.g. Figure 4 The discrete computing engine in the processor obtains the processing result corresponding to the input data.

[0162] Alternatively, the processing logic indicates that the input data corresponding to the first tensor is streaming data, which needs to be calculated in real time and includes information such as corresponding algorithms. Then, after obtaining the first tensor, the second processing module can instruct the computing engine of the second processing module itself (for example, Figure 4 A real-time computing engine in the processor obtains a processing result corresponding to the input data according to the first tensor and the processing logic.

[0163] It can be seen that in the embodiment of the present application, the user can configure the data processing method of the input data through the configuration interface of the first processing module, that is, configure the processing logic. Then, the first processing module transmits the configuration information to the second processing module, so that the second processing module is responsible for efficient underlying data processing and data processing operations such as real-time computing tasks. The second processing module may not be perceived by the user, that is, the user can interact with the data processing platform through the first processing module and input the processing logic, but the user does not need to perceive the specific processing process of the second processing module.

[0164] In addition, if Figure 4 In the example shown, the first processing module and the second processing module can also implement information interaction based on a shared memory.

[0165] Specifically, in some embodiments, the second processing module triggers the first processing module or the second processing module to obtain a processing result corresponding to the input data according to the first tensor and the processing logic, including:

[0166] The second processing module writes the first tensor into a storage space of the shared memory, where the storage space is used to store tensors including three dimensions;

[0167] The second processing module triggers the first processing module or the second processing module to read the first tensor from the storage space, so as to obtain a processing result corresponding to the input data according to the read first tensor and the processing logic.

[0168] In traditional technology, real-time processing frameworks such as Apache Flink are oriented to Java developers, while computing frameworks such as TensorFlow and PyTorch are implemented through Python. In traditional technology, although the real-time processing framework provides a Python application programming interface (API), the performance of the API is poor, which leads to performance bottlenecks for Python developers when using Apache Flink, and the efficiency of information interaction is usually low. In addition, in traditional technology, when information is exchanged between modules of different programming languages, it is usually necessary to convert the data to overcome the obstacles brought by different programming languages, and then store the converted data in a space such as a hard disk. Then, one of the modules needs to read the converted data from the space through more complex interaction instructions such as access instructions and data reading instructions before data parsing and processing can be performed. This traditional information interaction method is inefficient and requires multiple information interactions and data copies.

[0169] In the embodiment of the present application, through the shared memory, which is an efficient middleware, efficient information transmission between the first processing module and the second processing module can be performed based on the three-dimensional storage form.

[0170] Specifically, a storage space may be provided in the shared memory to store a tensor including three dimensions through the storage space.

[0171] After obtaining the first tensor, the second processing module can write the first tensor into the storage space of the shared memory. By reading data through the shared memory, multiple copies of data can be avoided and the processing efficiency of data can be improved. In addition, the first tensor is stored in the shared memory in the form of a tensor including three dimensions, so that the first processing module and the second processing module can quickly and easily identify and find the required data from the shared memory based on the information of each dimension, overcoming the information interaction barriers caused by different programming languages. It can be seen that through the shared memory, the data interaction between the first processing module and the second processing module about the first tensor can be realized, which can greatly improve the interaction performance between the first processing module and the second processing module, and overcome the information interaction barriers caused by different programming languages.

[0172] After the second processing module writes the first tensor into the storage space of the shared memory, the second processing module triggers the first processing module or the second processing module to read the first tensor from the storage space to obtain a processing result corresponding to the input data based on the read first tensor and the processing logic.

[0173] In the embodiment of the present application, the specific calculation method based on the read first tensor and the processing logic is not limited.

[0174] In one example, the second processing module may include a real-time computing engine applied to a real-time scenario, and the real-time computing engine may read the first tensor and other data from a shared memory, or directly obtain the first tensor and other data from a trigger or a preprocessing engine, etc., to achieve real-time computing and subsequent processing of the first tensor. The second programming language may adopt a programming language such as C++ with good performance, so as to improve the efficiency of real-time computing.

[0175] In another example, the data processing platform can be compatible with computing frameworks such as TensorFlow and PyTorch, and the real-time processing engine of the second processing module can be combined with the AI ​​model in the computing framework to perform data processing according to the first tensor. For example, the real-time processing engine of the second processing module implements the calculation of the first tensor, and then transmits the calculation result to the AI ​​model in the computing framework to perform operations such as feature extraction and feature processing.

[0176] In the above example, the second processing module can be used to implement operations such as processing and calculation of streaming data in real-time scenarios. Since the second processing module can use a programming language with better processing performance such as C++, and can perform real-time calculations quickly and efficiently based on the first tensor including three dimensions, low-latency processing efficiency of streaming data in real-time scenarios can be guaranteed.

[0177] In another example, the first processing module can perform data processing operations according to the first tensor and configuration information to obtain processing results corresponding to the input data. The first processing module can use python to implement an offline computing engine applied to offline scenarios, so that the first processing module can read the first tensor and other data from the shared memory, and batch process the read first tensor and other data through the offline computing engine to obtain processing results corresponding to the input data. The offline processing in the offline computing engine of the first processing module and the data preparation process for generating and storing the first tensor in the second processing module (for example, for executing the function of the preprocessing engine) can communicate asynchronously through shared memory, thereby achieving high-throughput data processing.

[0178] In other examples, the second processing module may include both a real-time computing engine applied to real-time scenarios and an offline computing engine applied to offline scenarios, thereby implementing data processing of offline scenarios and real-time scenarios through the second processing module.

[0179] 2. The configuration information includes preprocessing configuration information.

[0180] The preprocessing configuration information is used to configure one or more dimensions of the first tensor.

[0181] Step 302 includes:

[0182] The second processing module preprocesses the input data according to the preprocessing configuration information to obtain a first tensor.

[0183] The preprocessing configuration information is used to configure a preprocessing method for input data, wherein the preprocessing method may include information such as one or more dimensions of the first tensor.

[0184] Exemplarily, the preprocessing configuration information may include aggregation logic. For example, in a real-time scenario, real-time stream data is usually acquired one by one in sequence. Based on this, the preprocessing configuration information may include aggregation logic to describe the aggregation method of each piece of data in the real-time stream data.

[0185] For example, in one example, a piece of data is obtained every 2 seconds in the streaming data, and the user needs to perform calculations on the input data within every 20 seconds. The user can configure the aggregation logic in the preprocessing configuration information so that the preprocessing engine can aggregate the 10 pieces of data obtained every 20 seconds to generate a first tensor, and the time dimension of the first tensor can correspond to 20 seconds.

[0186] In some other examples, the preprocessing configuration information may also include other information, for example, configuration of the feature dimension and object dimension of the first tensor, a processing method when data is missing, etc.

[0187] In the embodiment of the present application, the second processing module may include a preprocessing engine, which may preprocess the input data according to the preprocessing configuration information to obtain the first tensor. The preprocessing engine may parse the input data, and then convert the parsed data into the form of the first tensor according to the preprocessing configuration information input by the user, and according to the preset configuration information in the second processing module.

[0188] It can be seen that in the embodiment of the present application, the user can configure the aggregation logic of the input data by pre-processing configuration information, so that the generated first tensor can meet the needs of subsequent calculation and processing, thereby reducing the processing steps and processing difficulty of the first tensor in the subsequent processing process.

[0189] 3. The configuration information includes trigger conditions.

[0190] The first processing module or the second processing module obtains a processing result corresponding to the input data according to the first tensor, including:

[0191] When it is determined based on the first tensor that the trigger condition is met, the second processing module obtains a processing result corresponding to the input data according to the first tensor.

[0192] In an embodiment of the present application, the trigger condition is used to indicate that the first tensor and other data required for subsequent data processing are ready, thereby indicating that the subsequent computing engine can process according to the first tensor and processing logic to obtain the processing result corresponding to the input data.

[0193] The specific content of the trigger condition can be in many situations and is not limited here.

[0194] In one example, the trigger condition may include a trigger condition based on time and / or count, such as Figure 4 In the example shown, the trigger condition can be implemented by a trigger in the second processing module, and the trigger can be understood as a functional module. In actual scenarios, the trigger and the preprocessing engine in the above embodiment can be different software modules, or they can be implemented by the same software module. This application does not impose any restrictions on this.

[0195] like Figure 5a In the example shown, the second processing module may include a trigger, and the storage space may implement unified processing of batch streams based on three-dimensional tensors (object, feature, time), where the maximum value of the object dimension t is N, the maximum value of the feature dimension m is M, and the maximum value of the time dimension t is T.

[0196] In this example, the trigger can sense the dimension information of the first tensor, and after determining based on the first tensor to obtain the first tensor within a specified time window (e.g., 20 seconds) or to obtain a specified number of tensors (the tensors include the first tensor), trigger the second processing module to obtain the required data (including the first tensor), thereby obtaining the processing result corresponding to the input data according to the required data and processing logic. The trigger condition can be a trigger condition for streaming data in a real-time scenario.

[0197] And in Figure 5bIn another example shown, the trigger condition may include a batch-based trigger condition. For example, the trigger may sense the first tensor to determine that after obtaining a specified batch of tensors (the tensors include the first tensor), the obtained specified batch of tensors is written to the shared memory to indicate that the specified batch of tensors is ready. In this example, the operation of the trigger writing the obtained specified batch of tensors to the shared memory can be considered as a trigger operation for the first processing module to obtain the processing result corresponding to the input data according to the first tensor and the processing logic. Then, the first processing module can obtain the processing result corresponding to the input data from the storage space of the shared memory according to the first tensor and the processing logic. The trigger condition can be a trigger condition for batch data in an offline scenario.

[0198] go through Figure 5a and Figure 5b After the data path, for both offline and real-time scenarios, the computing engine sees a tensor with three dimensions, the only difference is the size of the specified dimension of the tensor. For example, the value of object n in the real-time scenario can be 1<=n<=N, while the value of time t in the offline scenario is 1<=t<=T. Therefore, the above data path can shield the differences between real-time and offline scenarios and simplify the interface design of the computing engine.

[0199] It can be understood that in different application scenarios, the specific content of the configuration information can have a variety of situations. Based on the different needs of users, users can flexibly configure the processing process of the data processing method of any embodiment of the present application through one or more of the above-mentioned configuration information (such as processing logic, preprocessing configuration information, and trigger conditions). That is to say, it should be noted that the embodiments of different configuration information about processing logic, preprocessing configuration information, and trigger conditions can be combined with each other or implemented independently, and there is no limitation here. For example, the configuration information may include processing logic and trigger conditions, so that, based on the first tensor, when it is determined that the trigger condition is met, the second processing module can trigger the first processing module or the second processing module can obtain the processing result corresponding to the input data according to the first tensor and the processing logic.

[0200] In addition, in some embodiments, the storage and reading mechanism of the first tensor in the shared memory are improved.

[0201] The following is an exemplary introduction to the improvements of the storage and reading mechanism of the first tensor in the shared memory.

[0202] In one embodiment, the storage space includes a first storage subspace and a second storage subspace;

[0203] The second processing module writes the first tensor into the storage space of the shared memory, including:

[0204] The second processing module stores the first tensor into the first storage subspace based on the time information of the first tensor;

[0205] When the tensors within a specified time period in the time dimension in the first storage subspace are full, the second processing module stores the tensors within the specified time period in the second storage subspace, so that the first processing module and / or the second processing module reads the tensors within the specified time period from the second storage subspace.

[0206] In an embodiment of the present application, in order to facilitate timing calculations and save memory at the same time, the storage location can be adjusted in the shared memory based on the writing status of the tensor, so that the user can easily obtain the required timing window data and perform efficient calculations in the timing calculation, thereby greatly reducing the latency of the timing calculation.

[0207] Specifically, Figure 6 In the example shown, a first storage subspace and a second storage subspace may be determined in the storage space. The positions of the first storage subspace and the second storage subspace in the storage space are not limited. Figure 6 In the example shown, the first storage subspace can be located on the right side of the storage space, and the second storage subspace can be located on the left side of the storage space. The tensor (e.g., the first tensor) obtained based on the stream data can be continuously written into the first storage subspace based on the dimension information. When the first storage subspace is in a specified time period in the time dimension (e.g., Figure 6 When the tensor in the timing window W) shown in the figure is full, the tensor with the timing window W that was most recently written in the first storage subspace will be moved to the second storage subspace located on the left side of the storage space, so that the user can directly obtain the latest tensor with the timing window W from the second storage subspace and perform efficient timing calculations without spending a lot of time on data search in the storage space, thereby reducing the latency of timing calculations.

[0208] It should be noted that the first storage subspace and the second storage subspace may be different storage addresses of the same storage space to improve data transfer efficiency. Currently, in some other examples, the first storage subspace and the second storage subspace may also be located in different storage devices.

[0209] In one embodiment, a second tensor is also stored in the shared memory, and the second tensor includes two dimensions, namely, an object dimension and a feature dimension. In the second tensor, the feature value of each object is the latest feature value corresponding to the corresponding object, and the latest feature value is obtained by updating the second processing module according to the input data.

[0210] In the present application embodiment, Figure 7As shown, in order to support incremental calculation and more efficient cross-section calculation, a second tensor can be defined in shared memory. The second tensor can be understood as a cross-section state table, which is a two-dimensional tensor with the dimension of feature*object. In the second tensor, each feature of each object stores the latest value of the feature under the object. The second processing module can update the corresponding state value in the second tensor according to the latest acquired first tensor. At this time, the update can be performed in an in-place manner. The in-place method does not need to create a new data copy or additional storage space when modifying the data, but directly modifies the original data. In this way, it is convenient to quickly obtain the latest state value of the features of each object through the second tensor in scenarios such as incremental calculation or cross-section calculation, without the need to perform more complex and time-consuming data retrieval operations in shared memory, greatly improving the computational efficiency of scenarios such as incremental calculation and cross-section calculation.

[0211] Based on the above embodiments, it can be seen that based on the data and / or the second tensor in the storage space of the shared memory, the first processing module and / or the second processing module can quickly and conveniently read the required data and perform corresponding calculations in many application scenarios.

[0212] The data processing method of any of the above embodiments can be used not only in the AI ​​model reasoning stage based on input data, but also in the AI ​​model training stage. That is to say, the data processing platform that implements the data processing method can be an end-to-end feature calculation and model training reasoning platform.

[0213] In addition, in some embodiments, the AI ​​model training process can be optimized based on the data processing platform to provide users with an efficient AI model training method.

[0214] Below, an exemplary scheme for performing AI model training based on data in the storage space of shared memory and / or a second tensor is introduced.

[0215] In one embodiment, the data processing platform is used to train an artificial intelligence AI model;

[0216] The second processing module triggers the first processing module or the second processing module based on the trigger condition to obtain a processing result corresponding to the input data according to the first tensor and the processing logic, including:

[0217] At the nth moment, the second processing module triggers the first processing module to obtain the nth group of training data based on the content in the storage space at the nth moment in the shared memory and / or the second tensor at the nth moment, so that the first processing module performs the nth batch of training on the AI ​​model according to the nth group of training data, where n is a positive integer;

[0218] During the training process of the nth batch, at the n+1th moment, the second processing module triggers the first processing module to obtain the n+1th group of training data based on the content in the storage space at the n+1th moment in the shared memory and / or the second tensor at the n+1th moment, so that after the training of the nth batch is completed, the first processing module performs the n+1th batch of training on the AI ​​model according to the n+1th group of training data.

[0219] In the traditional AI model training process, usually all batch data can only be directly obtained at one time as training data, and training is performed based on all the batch data.

[0220] In the embodiments of the present application, considering that in real-time scenarios, the content of the storage space in the shared memory and the data of the second tensor can be updated with the real-time data, so that training data can be obtained from the storage space and / or the second tensor at different time nodes, and pipeline, multiple batches of training can be performed.

[0221] Specifically, data preparation can be achieved through the second processing module, and then the pipeline training of the AI ​​model can be achieved through the first processing module.

[0222] like Figure 8 The figure shows an exemplary schematic diagram of pipeline training of AI models.

[0223] The following is a detailed introduction to the pipeline training process formed by the stage corresponding to the nth moment and the stage corresponding to the n+1th moment. In actual application scenarios, the training process of the AI ​​model may include but is not limited to Figure 8 The training of the stage corresponding to the nth moment and the stage corresponding to the n+1th moment shown, that is, the stage corresponding to the nth moment can be any pipeline stage except the last pipeline stage in the training process of the AI ​​model.

[0224] like Figure 8 As shown, at the nth moment, the second processing module triggers the first processing module to obtain the nth group of training data based on the content in the storage space at the nth moment in the shared memory and / or the second tensor at the nth moment. The specific generation method of the nth group of training data is not limited here, and can be pre-configured by the user through configuration information.

[0225] It can be seen that in the embodiment of the present application, the data preparation operation required for the nth training is triggered at the nth time. An exemplary flow chart of the data preparation operation is as follows: Figure 9aAs shown. The sequence data obtained based on the batch data can be processed by the preprocessing engine to perform operations such as filtering, projection, aggregation, etc. to obtain the first tensor. Then, the trigger can detect whether the first tensor meets the batch-based trigger condition; if it does, if the shared memory is not full, the first tensor is written to the shared memory as the nth group of training data; if the shared memory is full, the data in the shared memory can be deleted and the first tensor is written to the shared memory as the nth group of training data.

[0226] Then, the first processing module may perform training according to the nth group of training data required for the nth training.

[0227] An exemplary schematic diagram of the training process is shown in Figure 9b As shown. The first processing module can determine whether there are n groups of training data prepared in the shared memory; if so, the nth group of training data is read from the shared memory by the CPU of the first processing module, and the nth group of training data is copied to the memory of the offline computing engine of the first processing module. Then the first processing module can perform the nth batch of training on the AI ​​model according to the nth group of training data. For example, forward propagation, back propagation, and gradient update of the AI ​​model can be performed. It can be seen that the data preparation process of the second processing module and the training process of the first processing module can communicate asynchronously through the shared memory.

[0228] In addition, in an embodiment of the present application, during the training process of the nth batch, the data preparation operation required for the n+1th training can be triggered. Specifically, during the training process of the nth batch, at the n+1th moment, the second processing module triggers the first processing module to obtain the n+1th group of training data based on the content in the storage space at the n+1th moment in the shared memory and / or the second tensor at the n+1th moment, so that after the training of the nth batch is completed, the first processing module performs the n+1th batch of training on the AI ​​model according to the n+1th group of training data.

[0229] During the pipeline training process, the time for data preparation is less than the time required for the first processing module to perform a batch training. In this way, the data preparation time can be covered by the training process of the previous stage without adding additional processing time, so that AI model training can be carried out continuously and uninterruptedly without waiting for data, thereby achieving maximum throughput and improving training efficiency.

[0230] Of course, the embodiments of the present application can also be used for AI model training in offline scenarios. Based on the data processing flow of the embodiments of the present application, the second processing module can store the tensors corresponding to the batch data into the shared memory in batches, so that the first processing module can read the corresponding tensors from the shared memory in batches and perform pipeline training.

[0231] In some examples, the above process can also be trained in a distributed parallel mode. Fig.10 As shown, an exemplary distributed parallel mode can be to start multiple training processes using a computing framework (such as PyTorch). Gradient synchronization can be performed between these training processes, and there can be many situations for the AI ​​models of different training processes, for example, different submodules of the AI ​​model (such as different expert models), or different AI models, etc. Fig.10 In the example shown, each training process can be paired with a data preparation process to ensure the uninterrupted operation of the training task, thereby further improving the training efficiency.

[0232] In some other examples, the embodiments of the present application can also be used for data processing in real-time scenarios.

[0233] In the real-time phase, the main difference between triggers and those in the training phase is that they can be time-based and count-based.

[0234] like Fig.11 The figure shows an exemplary flow chart of reasoning based on input data in a real-time scenario.

[0235] in, Fig.11 The preprocessing operations shown are similar to Figure 9a The preprocessing operations shown may be the same, for example, may include filtering, projection, and aggregation, and the data processing flow (for example, including feature calculation and model inference) also corresponds to the feature calculation and model training processes of the training process, so that users do not need to perform secondary code development.

[0236] Users can flexibly configure trigger parameters in Python according to actual scenario requirements, so that the online feature calculation and reasoning process can better adapt to different real-time scenarios, thereby improving the overall computing performance and response speed.

[0237] Distributed parallel processing can also be performed in the real-time processing stage to further improve the data processing speed and system throughput. For example, first, the real-time input data is segmented according to a certain strategy (for example, according to data type, data source, etc.). Then, the input data can be evenly distributed to the computing nodes of multiple computing node clusters to achieve parallel processing of data. On each computing node, the real-time data will be filtered, projected, and aggregated by the preprocessing engine, and then enter the trigger. The trigger passes the data to the real-time computing engine after the trigger conditions are met based on the latency and throughput requirements. In this example, the real-time calculation process can be performed in a second processing module with better performance. For example, when the second processing module is implemented based on C++, the real-time calculation process can be performed in a C++ process to achieve low-latency calculation.

[0238] It can be seen that in the data processing method of the embodiment of the present application, through flexible configuration content, it can ensure both high throughput performance in offline scenarios and low latency performance in real-time scenarios. That is to say, the data processing platform that implements the data processing method is a batch-stream integrated computing framework that can meet the needs of offline scenarios that require high throughput performance to improve data processing speed, and can meet the needs of real-time scenarios that require low latency performance.

[0239] The data processing method can be applied to a variety of application scenarios, and each application scenario is introduced with examples below.

[0240] 1. Financial field: The data processing method of any embodiment of the present application can be applied to financial scenarios such as financial risk control, credit scoring, user profiling, and quantitative trading, and the input data can be financial data. The use of a batch-stream integrated data processing platform can quickly process a large amount of real-time transaction data and offline historical data, improve the efficiency of AI model training and reasoning, thereby improving the accuracy of risk control and credit assessment, and facilitate the real-time adjustment and optimization of quantitative trading strategies. Among them, in real-time scenarios, by analyzing financial market data in real time, quantitative trading strategies are obtained, which can quickly respond to market changes and realize high-frequency trading and automated investment management.

[0241] 2. Internet of Things (IoT): The data processing method of any embodiment of the present application can be applied to the field of IoT, such as smart home, industrial automation and other scenarios. By utilizing a batch-stream integrated data processing platform, efficient processing and analysis of a large amount of real-time data and historical data can be achieved, providing support for intelligent decision-making and improving the efficiency of equipment management and optimization.

[0242] 3. E-commerce platform: The data processing method of any embodiment of this application can be applied to the recommendation system, advertising system and other scenarios of the e-commerce platform. By using the batch-stream integrated data processing platform, a large amount of real-time user behavior data and offline product data can be quickly processed to improve the effect of recommendation algorithms and advertising, thereby improving user experience and platform revenue.

[0243] 4. Intelligent manufacturing: The data processing method of any embodiment of the present application can be applied to the field of intelligent manufacturing, such as quality control of production lines, equipment maintenance and other scenarios. By using a batch-stream integrated data processing platform, efficient processing and analysis of real-time production data and offline historical data can be achieved to provide support for production decisions and improve production efficiency and product quality.

[0244] The data processing method provided in the embodiments of the present application is described above from multiple aspects. The data processing device provided in the embodiments of the present application is described below in conjunction with the accompanying drawings.

[0245] like Fig.12As shown, an embodiment of the present application provides a data processing device 120, which is applied to a data processing platform, and the data processing platform includes a first processing module 1201 and a second processing module 1202, wherein the first processing module 1201 provides a configuration interface, and the configuration interface is used to instruct a user to input configuration information. The first processing module 1201 is implemented based on a first programming language, and the configuration information is described based on the first programming language. The second processing module 1202 is implemented based on a second programming language, and the second programming language is different from the first programming language.

[0246] The first processing module 1201 or the second processing module 1202 is used to obtain input data, where the input data is stream data or sequence data obtained based on batch data;

[0247] The second processing module 1202 is used to pre-process the input data to obtain a first tensor, where the first tensor includes three dimensions, namely, an object dimension corresponding to an object in the input data, a feature dimension corresponding to a feature of the object, and a time dimension;

[0248] The first processing module 1201 or the second processing module 1202 is used to obtain a processing result corresponding to the input data according to the first tensor.

[0249] Optionally, the first processing module 1201 is used to:

[0250] receiving configuration information, the configuration information including processing logic;

[0251] Passing processing logic to the second processing module 1202;

[0252] The second processing module 1202 is used to trigger the first processing module 1201 or the second processing module 1202 to process the first tensor according to the processing logic to obtain a processing result corresponding to the input data.

[0253] Optionally, the second processing module 1202 is used to:

[0254] Write the first tensor into the storage space of the shared memory, where the storage space is used to store tensors including three dimensions;

[0255] The first processing module 1201 or the second processing module 1202 is triggered to read the first tensor from the storage space, so as to obtain a processing result corresponding to the input data according to the read first tensor and the processing logic.

[0256] Optionally, the configuration information includes a trigger condition;

[0257] The second processing module 1202 is used for:

[0258] When it is determined based on the first tensor that the trigger condition is satisfied, the second processing module 1202 obtains a processing result corresponding to the input data according to the first tensor.

[0259] Optionally, the configuration information includes preprocessing configuration information, where the preprocessing configuration information is used to configure one or more dimensions of the first tensor;

[0260] The second processing module 1202 is used for:

[0261] Preprocess the input data according to the preprocessing configuration information to obtain the first tensor.

[0262] Optionally, the storage space includes a first storage subspace and a second storage subspace;

[0263] The second processing module 1202 is used for:

[0264] Based on the time information of the first tensor, storing the first tensor in a first storage subspace;

[0265] When the tensors within a specified time period in the time dimension in the first storage subspace are full, the tensors within the specified time period are stored in the second storage subspace, so that the first processing module 1201 and / or the second processing module 1202 read the tensors within the specified time period from the second storage subspace.

[0266] Optionally, a second tensor is also stored in the shared memory, and the second tensor includes two dimensions, namely, an object dimension and a feature dimension. In the second tensor, the feature value of each object is the latest feature value corresponding to the corresponding object, and the latest feature value is obtained by updating the second processing module 1202 according to the input data.

[0267] Optionally, the data processing platform is used to train the artificial intelligence AI model;

[0268] The second processing module 1202 is used for:

[0269] At the nth moment, the first processing module 1201 is triggered to obtain the nth group of training data based on the content in the storage space at the nth moment in the shared memory and / or the second tensor at the nth moment, so that the first processing module 1201 performs the nth batch of training on the AI ​​model according to the nth group of training data, where n is a positive integer;

[0270] During the training process of the nth batch, at the n+1th moment, the first processing module 1201 is triggered to obtain the n+1th group of training data based on the content in the storage space at the n+1th moment in the shared memory and / or the second tensor at the n+1th moment, so that after the training of the nth batch is completed, the first processing module 1201 performs the n+1th batch of training on the AI ​​model according to the n+1th group of training data.

[0271] Optionally, the data processing device 120 is applied in the financial field, and the input data includes financial data.

[0272] In an embodiment of the present application, a module is used as an example of a software functional unit, and a data processing device may include code running on a computing instance. Among them, the computing instance may be at least one of a physical host (computing device), a virtual machine, a container and other computing devices. Furthermore, the above-mentioned computing device may be one or more. For example, the data processing device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same available zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0273] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway must be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0274] As an example of a hardware functional unit, a data processing device may include at least one computing device, such as a server, etc. Alternatively, the data processing device may also be a device implemented using an application specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex PLD (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0275] The multiple computing devices included in the data processing device can be distributed in the same region or in different regions. The multiple computing devices included in the data processing device can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the data processing device can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0276] It should be noted that, in other embodiments, the first processing module 1201 and the second processing module 1202 can be used to execute any step in the data processing method, and the steps that the first processing module 1201 and the second processing module 1202 are responsible for implementing can be specified as needed. The first processing module 1201 and the second processing module 1202 respectively implement different steps in the data processing method to realize the full functions of the data processing device.

[0277] The present application embodiment also provides a computing device 130. Fig.13 As shown, the computing device 130 includes: a bus 132, a processor 134, a memory 136, and a communication interface 138. The processor 134, the memory 136, and the communication interface 138 communicate with each other through the bus 132. The computing device 130 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 130.

[0278] The bus 132 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 The bus 134 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 134 may include a path for transmitting information between various components of the computing device 130 (eg, the memory 136, the processor 134, and the communication interface 138).

[0279] The processor 134 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0280] The memory 136 may include a volatile memory, such as a random access memory (RAM). The processor 134 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0281] The memory 136 stores executable program codes, and the processor 134 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module and training module, thereby realizing the data processing method applied to the computing device cluster in the above-mentioned embodiment, that is, the memory 136 stores instructions for executing the data processing method applied to the computing device cluster in the above-mentioned embodiment.

[0282] The communication interface 138 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 130 and other devices or a communication network.

[0283] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0284] like Fig.14 As shown, the computing device cluster includes at least one computing device 130. The memory 136 in one or more computing devices 130 in the computing device cluster may store the same instructions for executing the data processing method.

[0285] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 130 may jointly execute instructions for executing the data processing method.

[0286] It should be noted that the memory 136 in different computing devices 130 in the computing device cluster may store different instructions, respectively used to execute part of the functions of the data processing method. That is, the instructions stored in the memory 136 in different computing devices 130 may implement one or more functions of the acquisition module.

[0287] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.15 A possible implementation is shown. Fig.15 As shown, two computing devices 130A and 130B are connected via a network. Specifically, the connection to the network is made through the communication interface in each computing device. In this type of possible implementation, the memory 136 in the computing device 130A may store instructions for executing the functions of the first processing module. At the same time, the memory 136 in the computing device 130B may store instructions for executing the functions of the second processing module. Alternatively, the memory 136 in the computing device 130A may store instructions for executing part of the functions of the second processing module. At the same time, the memory 136 in the computing device 130B may store instructions for executing another part of the functions of the second processing module, etc.

[0288] It should be understood that Fig.15 The functionality of computing device 130A shown in FIG. 1 may also be performed by multiple computing devices 130. Similarly, the functionality of computing device 130B may also be performed by multiple computing devices 130.

[0289] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.14 and Fig.15 Connection mode of computing device cluster. The difference is that the memory 136 in one or more computing devices 130 in the computing device cluster may store the same instructions for executing the data processing method.

[0290] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 130 may jointly execute instructions for executing the data processing method.

[0291] It should be noted that the memory 136 in different computing devices 130 in the computing device cluster may store different instructions for executing partial functions of the data processing method. That is, the instructions stored in the memory 136 in different computing devices 130 may implement one or more functions of the processing module.

[0292] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method.

[0293] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method.

[0294] The embodiment of the present application also provides a chip system, which includes a processor, and the processor is used to implement the steps performed by the above-mentioned computing device cluster. In a possible design, the chip system may also include a memory, and the memory is used to store necessary program instructions and data. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0295] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0296] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0297] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0298] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0299] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc. Various media that can store program codes.

Claims

1. A data processing method, characterized in that: The method is applied to a data processing platform, the data processing platform includes a first processing module and a second processing module, wherein the first processing module provides a configuration interface, the configuration interface is used to instruct a user to input configuration information, the first processing module is implemented based on a first programming language, the configuration information is described based on the first programming language, the second processing module is implemented based on a second programming language, the second programming language is different from the first programming language, and the method includes: The first processing module or the second processing module obtains input data, where the input data is stream data or batch data; The second processing module preprocesses the input data to obtain a first tensor, where the first tensor includes three dimensions, the three dimensions being an object dimension corresponding to an object in the input data, a feature dimension corresponding to a feature of the object, and a time dimension; The first processing module or the second processing module obtains a processing result corresponding to the input data according to the first tensor.

2. The method according to claim 1, characterized in that The method further comprises: The first processing module receives the configuration information, where the configuration information includes processing logic; The first processing module transfers the processing logic to the second processing module; The first processing module or the second processing module obtains a processing result corresponding to the input data according to the first tensor, including: The second processing module triggers the first processing module or the second processing module processes the first tensor according to the processing logic to obtain a processing result corresponding to the input data.

3. The method according to claim 2, characterized in that The second processing module triggers the first processing module or the second processing module obtains a processing result corresponding to the input data according to the first tensor and the processing logic, including: The second processing module writes the first tensor into a storage space of a shared memory, where the storage space is used to store tensors including the three dimensions; The second processing module triggers the first processing module or the second processing module reads the first tensor from the storage space to obtain a processing result corresponding to the input data according to the read first tensor and the processing logic.

4. The method according to claim 2 or 3, characterized in that: The configuration information includes a trigger condition; The first processing module or the second processing module obtains a processing result corresponding to the input data according to the first tensor, including: When it is determined based on the first tensor that the trigger condition is met, the second processing module obtains a processing result corresponding to the input data according to the first tensor.

5. The method according to any one of claims 2 to 4, characterized in that: The configuration information includes preprocessing configuration information, where the preprocessing configuration information is used to configure one or more dimensions of the first tensor; The second processing module preprocesses the input data to obtain a first tensor, including: The second processing module preprocesses the input data according to the preprocessing configuration information to obtain the first tensor.

6. The method according to claim 3, characterized in that The storage space includes a first storage subspace and a second storage subspace; The second processing module writes the first tensor into a storage space of a shared memory, including: The second processing module stores the first tensor into the first storage subspace based on the time information of the first tensor; When the first storage subspace is full of tensors within a specified time period on the time dimension, the second processing module stores the tensors within the specified time period in the second storage subspace, so that the first processing module and / or the second processing module reads the tensors within the specified time period from the second storage subspace.

7. The method according to claim 3 or 6, characterized in that: The shared memory also stores a second tensor, which includes two dimensions, namely the object dimension and the feature dimension. In the second tensor, the feature value of each object is the latest feature value corresponding to the corresponding object, and the latest feature value is updated by the second processing module according to the input data.

8. The method according to claim 7, characterized in that The data processing platform is used to train the artificial intelligence AI model; The second processing module triggers the first processing module or the second processing module reads the first tensor from the storage space to obtain a processing result corresponding to the input data according to the read first tensor and the processing logic, including: At the nth moment, the second processing module triggers the first processing module to obtain the nth group of training data based on the content in the storage space at the nth moment in the shared memory and / or the second tensor at the nth moment, so that the first processing module performs the nth batch of training on the AI ​​model according to the nth group of training data, where n is a positive integer; During the training process of the nth batch, at the n+1th moment, the second processing module triggers the first processing module to obtain the n+1th group of training data based on the content in the storage space at the n+1th moment in the shared memory and / or the second tensor at the n+1th moment, so that after the training of the nth batch is completed, the first processing module performs the n+1th batch of training on the AI ​​model according to the n+1th group of training data.

9. The method according to any one of claims 1 to 8, characterized in that: The data processing method is applied in the financial field, and the input data includes financial data.

10. A data processing device, characterized in that: Applied to a data processing platform, the data processing platform comprises a first processing module and a second processing module, wherein the first processing module provides a configuration interface, the configuration interface is used to instruct a user to input configuration information, the first processing module is implemented based on a first programming language, the configuration information is described based on the first programming language, and the second processing module is implemented based on a second programming language, and the second programming language is different from the first programming language; The first processing module or the second processing module is used to obtain input data, where the input data is stream data, or is sequence data obtained based on batch processing data; The second processing module is used to preprocess the input data to obtain a first tensor, where the first tensor includes three dimensions, the three dimensions being an object dimension corresponding to an object in the input data, a feature dimension corresponding to a feature of the object, and a time dimension; The first processing module or the second processing module is used to obtain a processing result corresponding to the input data according to the first tensor.

11. The device according to claim 10, characterized in that The first processing module is used for: receiving the configuration information, the configuration information including processing logic; transferring the processing logic to the second processing module; The second processing module is used to: trigger the first processing module or the second processing module to process the first tensor according to the processing logic to obtain a processing result corresponding to the input data.

12. The device according to claim 11, characterized in that The second processing module is used for: Writing the first tensor into a storage space of a shared memory, where the storage space is used to store tensors including the three dimensions; The first processing module or the second processing module is triggered to read the first tensor from the storage space, so as to obtain a processing result corresponding to the input data according to the read first tensor and the processing logic.

13. The device according to claim 11 or 12, characterized in that The configuration information includes a trigger condition; The second processing module is used for: When it is determined based on the first tensor that the trigger condition is met, the second processing module obtains a processing result corresponding to the input data according to the first tensor.

14. The device according to any one of claims 11 to 13, characterized in that: The configuration information includes preprocessing configuration information, where the preprocessing configuration information is used to configure one or more dimensions of the first tensor; The second processing module is used for: The input data is preprocessed according to the preprocessing configuration information to obtain the first tensor.

15. The device according to claim 12, characterized in that The storage space includes a first storage subspace and a second storage subspace; The second processing module is used for: Based on the time information of the first tensor, storing the first tensor in the first storage subspace; When the tensors within the specified time period on the time dimension are full in the first storage subspace, the tensors within the specified time period are stored in the second storage subspace, so that the first processing module and / or the second processing module reads the tensors within the specified time period from the second storage subspace.

16. The device according to claim 12 or 15, characterized in that The shared memory also stores a second tensor, which includes two dimensions, namely the object dimension and the feature dimension. In the second tensor, the feature value of each object is the latest feature value corresponding to the corresponding object, and the latest feature value is updated by the second processing module according to the input data.

17. The device according to claim 16, characterized in that The data processing platform is used to train the artificial intelligence AI model; The second processing module is used for: At the nth moment, trigger the first processing module to obtain the nth group of training data based on the content in the storage space at the nth moment in the shared memory and / or the second tensor at the nth moment, so that the first processing module performs the nth batch of training on the AI ​​model according to the nth group of training data, where n is a positive integer; During the training process of the nth batch, at the n+1th moment, the first processing module is triggered to obtain the n+1th group of training data based on the content in the storage space at the n+1th moment in the shared memory and / or the second tensor at the n+1th moment, so that after the training of the nth batch is completed, the first processing module performs the n+1th batch of training on the AI ​​model according to the n+1th group of training data.

18. The device according to any one of claims 10 to 17, characterized in that: The data processing device is applied in the financial field, and the input data includes financial data.

19. A computing device, characterized in that include: at least one memory for storing a program; at least one processor, configured to execute the program stored in the memory; Wherein, when the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1-9.

20. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-9.

21. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on a processor, the processor is caused to execute the method according to any one of claims 1 to 9.

22. A computer program product, characterized in that When the computer program product runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 9.

Images