Data production method and device, equipment and storage medium

Abstract

The invention discloses a data production method, device and equipment and a storage medium, and relates to the technical field of data processing, the method is applied to a data management platform, and the method comprises the following steps: receiving a data production demand of a user; creating a first data production task based on the data production demand and a preset data version list; creating a data scheduling queue based on the first data production task; and performing data production based on the machine cluster, a preset scheduling algorithm and the data scheduling queue to obtain a data production result. By means of the method, rapid production of large-scale data sets can be achieved, and meanwhile multiple data versions are set to meet different user requirements.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data production method, device, equipment and storage medium. Background Art

[0002] Existing data production methods often solve how to perform data mining on large data sets to obtain production data, but cannot achieve rapid iterative updates of production data.

[0003] Therefore, existing data production methods have the problem of being unable to iterate and update quickly when producing data for large data sets.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a data production method, device, equipment and storage medium, aiming to solve the technical problem that existing data production methods cannot quickly iterate and update when producing large data sets.

[0006] To achieve the above purpose, the present application proposes a data production method, which is applied to a data management platform, the data management platform includes a machine cluster, and the method includes: Receive data production requirements from users; Based on the data production requirement and the preset data version list, create a first data production task; Based on the first data production task, create a data scheduling queue; Based on the machine cluster, the preset scheduling algorithm and the data scheduling queue, data production is performed to obtain data production results.

[0007] In one embodiment, the first data production task includes target data version information and target original data information, the first data production task type is determined by the data production requirement, the first data production task type includes a first type and a second type, and the step of creating the first data production task based on the data production requirement and the preset data version list includes: When the first data production task type is the first type, determining target data version information according to data production requirements and a preset data version list, and determining target original data according to data production requirements and a preset scheduling algorithm to obtain target original data information; When the first data production task type is the second type, determining target data version information and target original data information according to data production requirements and a preset data version list; Based on the target data version information and the target original data information, a first data production task is created.

[0008] In one embodiment, before the step of receiving the user's data production demand, the method further includes: Receive data version requirements from users; Based on the data version requirement, determine version information of the data version; Creating a data version according to the version information; Based on the data version, the data version list is generated.

[0009] In one embodiment, the version information includes version characteristics, and after the step of generating the data version list based on the data version, the step further includes: If the version characteristic of the data version is a preset characteristic, automatically creating a second data production task based on the data version; Based on the second data production task, confirm the target original data and task priority; Based on the target original data and the task priority, a data scheduling queue is created, and the following steps are performed: based on the machine cluster, the preset scheduling algorithm and the data scheduling queue, data production is performed to obtain a data production result; If the version characteristic of the data version is not a preset characteristic, the step of receiving a user's data production demand is executed.

[0010] In one embodiment, the step of performing data production based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain the data production result includes: Using a preset scheduling algorithm, regularly detecting the resource status of the machine cluster according to a preset period; According to the resource status, scheduling target original data from the data scheduling queue to obtain target data information; Sending the target data information to the machine cluster; Based on the machine cluster, data production is performed according to the target data information to obtain a data production result.

[0011] In one embodiment, the target data information includes the target data version information, the target data version information includes an associated branch algorithm, and the step of performing data production based on the machine cluster and the target data information to obtain a data production result includes: Based on the machine cluster, configure and deploy containers according to the target data information; Based on the container, target data is acquired, and data production is performed on the target data according to the association branch algorithm to obtain a data production result.

[0012] In one embodiment, the method further comprises: Retrieving and managing the data versions according to a preset data version list; Managing the data production tasks according to the data scheduling queue; The data production results are uploaded and stored in the cloud database of the data management platform, and the intermediate output data on the machine cluster is deleted.

[0013] In addition, to achieve the above purpose, the present application also proposes a data production device, which is applied to a data management platform, the data management platform includes a machine cluster, and the data production device includes: A receiving module, used to receive data production requirements from users; A task creation module, used to create a first data production task based on the data production requirement and a preset data version list; A queue creation module, used to create a data scheduling queue based on the first data production task; The data production module is used to produce data based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain data production results.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a data production device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the data production method described above.

[0015] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data production method described above are implemented.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: The data production method, apparatus, device and storage medium proposed in the embodiments of the present application specifically receive data production requirements from users; create a first data production task based on the data production requirements and a preset data version list; create a data scheduling queue based on the first data production task; and perform data production based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain a data production result.

[0017] This application receives the user's data production needs, creates a data production task in combination with a preset data version list, then creates a data production scheduling queue based on the target data version and target original data in the data production task, and then uses a machine cluster and a preset scheduling algorithm to produce data based on the data scheduling queue, thereby completing the rapid production of large-scale data sets. At the same time, it meets different user needs based on multiple data versions in the data version list, and enables any multiple versions of data to be scheduled and processed at the same time, ensuring the diversity of production data and rapid iterative updates. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the data production method of this application; Figure 2 A schematic diagram of the process flow provided for the second embodiment of the data production method of this application; Figure 3 A schematic diagram of the process provided for the third embodiment of the data production method of this application; Figure 4 A schematic diagram of a process flow provided for Embodiment 4 of the data production method of this application; Figure 5 A schematic diagram of the process flow provided for Example 5 of the data production method of this application; Figure 6 This is an example diagram of a page where a user selects raw data according to a time range involved in the second embodiment of the present application; Figure 7 This is an example diagram of a page for user inputting data version requirements involved in the fourth embodiment of the present application; Figure 8 This is an example diagram of the management page of the data scheduling queue involved in Example 5 of the present application; Fig. 9 This is a schematic diagram of the module structure of the data production device according to an embodiment of the present application; Fig.10 A schematic diagram of the device structure of the hardware operating environment involved in the data production method in the embodiment of the present application.

[0021] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0023] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of the embodiment of the present application is: receiving the user's data production requirements; creating a first data production task based on the data production requirements and a preset data version list; creating a data scheduling queue based on the first data production task; performing data production based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain data production results.

[0025] Technical terms involved in the embodiments of this application: Container: Containers generally refer to a lightweight, portable, and software packaging technology that is independent of other applications and the host machine. Specifically, container technology allows developers to package an application and all its dependencies into an independent container, which can run in any environment that supports container operation without any modification or reconfiguration.

[0026] Container technology allows multiple containers to run on the same physical or virtual machine by providing lightweight virtualization. Each container has its own independent file system, process space, and network stack, thereby achieving efficient isolation and utilization of resources. The container image contains the application and all its dependencies, which allows the container to be seamlessly migrated and run in different environments, whether it is a development environment, a test environment, or a production environment. In a machine cluster, containers can be easily scaled horizontally to meet growing load requirements by adding more container instances.

[0027] Existing data production methods often focus on how to conduct data mining on large-scale data sets to produce data that meets the needs. They do not take into account the diverse needs of users and are unable to achieve the rapid production of multiple versions of production data. As a result, there is a technical problem of being unable to quickly iterate and update when producing data for large-scale data sets.

[0028] The present application provides a solution, which receives the data production needs of users, creates data production tasks in combination with a preset data version list, and then creates a data production scheduling queue based on the target data version and target original data in the data production task. Then, through a machine cluster and a preset scheduling algorithm, data production is performed according to the data scheduling queue, thereby completing the rapid production of large-scale data sets. At the same time, different user needs are met based on multiple data versions in the data version list, so that any multiple versions of data can be scheduled and processed at the same time, ensuring the diversity of production data and rapid iterative updates.

[0029] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a data production device that can realize the above functions, a data management platform, etc. The following takes the data management platform as an example to illustrate this embodiment and the following embodiments.

[0030] Based on this, the present application embodiment provides a data production method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data production method of this application.

[0031] In this embodiment, the method is applied to a data management platform, the data management platform includes a machine cluster, and the data production method includes steps S110 to S140: Step S110, receiving the data production demand of the user; Specifically, the data management platform first needs to receive the user's data production needs and understand the importance of data production, target data characteristics, and usage. The data management platform usually receives the user's data production needs through multiple form components in a visual page. Among them, the user's data production needs often include information such as the source of original data, production data type and format, original data range and granularity, production data quality requirements, and the importance of data production.

[0032] The scope of raw data refers to the time range, geographical range, business scope, etc. of the data that users need to clarify; the granularity requirement of raw data refers to the specific calculation unit of raw data that users need to determine, such as days or hours. The quality requirement of production data refers to the user's requirements for the quality of production data, such as data accuracy, completeness, consistency, timeliness, etc.

[0033] It should be understood that the user's data production requirements may also include a delivery time requirement for production data, which may be used to characterize the importance of data production.

[0034] Step S120, creating a first data production task based on the data production requirement and a preset data version list; It should be noted that the preset data version list refers to a list of several data versions created in advance by developers based on historical user data version requirements. The data version is used to distinguish different identifiers of production data caused by different processed data sets, data processing models, data production methods, etc. Usually, different data versions correspond to the production of different types of data sets.

[0035] Specifically, based on the received user data production requirements, the relevant information of the corresponding target data version is determined from the preset data version list, including but not limited to the version name, organization, version type, data version effective time, data version expiration time and other information of the target data version.

[0036] The version name is used to identify different data versions. The version name can be Chinese, numbers, letters, or a combination thereof. The affiliated organization is the team to which the data version belongs, which is the organization that has the need to use the data version. The version type is used to characterize the purpose of the production data corresponding to the data version. For example, the version type can be used to distinguish whether the production data belongs to the training data or evaluation data of a certain algorithm or model. The effective time and expiration time of the data version can be used to determine the timeliness of the data version.

[0037] It can be understood that different organizations to which data versions belong can also be used to distinguish the production of different functional model data.

[0038] Then, the target original data information is obtained according to the user's data production needs, and the priority level of the data production task is determined according to its importance, and then the first data production task is created in combination with the relevant information of the target data version. Among them, the target original data information refers to the data source, data time range, data magnitude, data format, data content, data quality description and other information of the original data required by the user for data production.

[0039] Step S130, creating a data scheduling queue based on the first data production task; It should be noted that the preset scheduling algorithm refers to the algorithm that the developer pre-sets in the data management platform for real-time scheduling of production data. The preset scheduling algorithm can be a polling algorithm, a least connection algorithm, a random algorithm, etc.

[0040] Specifically, according to the first data production task created, the order of the data production tasks is confirmed according to the task generation time, and the order can also be determined according to the target original data byte amount associated with the data production task. Then, the corresponding target original data is arranged in the above order, so as to construct a data scheduling queue. It should be understood that the target original data carries the target data version information.

[0041] In this embodiment, it is preferred to sort the data production tasks first according to the priority of the first data production task, and secondly according to the creation time of the first data production task, that is, the time when the target original data is added to the data scheduling queue.

[0042] Step S140, based on the machine cluster, the preset scheduling algorithm and the data scheduling queue, data production is performed to obtain a data production result.

[0043] Then, based on the preset scheduling algorithm, the target original data to be processed is scheduled from the data scheduling queue, and then the target original data to be processed is distributed to multiple servers in the machine cluster that can support processing, and then data production is performed according to the corresponding target data version information to obtain the data production result.

[0044] In a feasible implementation manner, step 140 may include steps B01 to B04: Step B01, using a preset scheduling algorithm, regularly detecting the resource status of the machine cluster according to a preset period; Step B02, according to the resource status, scheduling target original data from the data scheduling queue to obtain target data information; Step B03, sending the target data information to the machine cluster; In this implementation, the preset scheduling algorithm is to regularly detect the resource status of the machine cluster according to a preset period, including information such as available resources, used resources, and corresponding data production tasks being executed, so as to subsequently reasonably schedule resources for data production based on the resource status, avoid frequent scheduling and detection of machines, and reduce the occurrence rate of machine abnormalities.

[0045] Among them, the preset period is a detection time period pre-set by the developer based on a comprehensive judgment of the performance of the machine cluster, the magnitude of common raw data, the historical data production time, and the actual data production needs. In this embodiment, the preset period can be 5 minutes.

[0046] Then, according to the available resources and idle servers in the resource status of the machine cluster, the data processing volume that can be scheduled by the current available resources is calculated, and then according to the schedulable data processing volume, the original data to be processed is confirmed from the data scheduling queue in the queue order, and the target data information is generated based on the original data to be processed, and the target data information is distributed to the machine cluster. Among them, the target data information includes information related to the original data to be processed, the amount of data to be processed, the target data version information, etc.

[0047] Step B04: Based on the machine cluster, data production is performed according to the target data information to obtain a data production result.

[0048] Specifically, the data management platform uses a machine cluster to download and obtain the original data to be processed and the data processing flow corresponding to the original data according to the target data information, and produces data for the original data according to the corresponding data processing flow to obtain data production results.

[0049] In one implementation, the target data information includes the target data version information, the target data version information includes an associated branch algorithm, and the step B04 includes steps C01-C02: Step C01, based on the machine cluster, configure and deploy the container according to the target data information; Step C02, based on the container, obtain target data, and perform data production on the target data according to the association branch algorithm to obtain a data production result.

[0050] It should be noted that the associated branch algorithm is a branch algorithm or code used to determine the data corresponding to the production data version. Usually, the data production of a data version is implemented by a set of codes or algorithms. According to different branch algorithms, data preprocessing tasks, data integration, data mining tasks, data analysis tasks, etc. can be implemented while producing corresponding data.

[0051] Specifically, the data management platform uses container technology to pre-deploy multiple containers in the machine cluster to efficiently utilize resources. Then, based on the target data information obtained, further obtain the data source, data analysis requirements, data production requirements, data volume and other information of the original data to be processed. Then, based on the data source, data analysis requirements, data production requirements, data volume and other information, the container in the machine cluster dynamically configures the data analysis format, data access interface and other information to the container environment, and limits the container's CPU, memory space and other resources based on the data volume, and then restarts the container to implement the container deployment.

[0052] Finally, the container needs to obtain the original data to be processed, that is, the target data, from the database of the data management platform, and obtain the branch algorithm corresponding to the target data version from the database of the data management platform based on the target data version information. The branch algorithm corresponding to the target data version is used to perform a series of data production operations such as data parsing, data mining, effective data extraction, and effective data analysis on the original data to be processed. The specific implementation function is determined based on the branch algorithm associated with the target data version.

[0053] In this implementation, the resource status of the machine cluster is detected through a preset scheduling algorithm, and then reasonable resource scheduling is implemented for data production based on the resource status, thereby avoiding frequent scheduling and detection of machines and reducing the incidence of machine abnormalities.

[0054] This embodiment provides a data production method, which includes receiving a user's data production demand; creating a first data production task based on the data production demand and a preset data version list; creating a data scheduling queue based on the first data production task; and performing data production based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain a data production result.

[0055] This application receives the data production needs of users, creates data production tasks in combination with a preset data version list, then creates a data production scheduling queue based on the target data version and target original data in the data production task, and then uses a machine cluster and a preset scheduling algorithm to produce data based on the data scheduling queue. While creatively designing the queue scheduling function to give full play to the performance of the machine cluster, it completes the rapid production of large-scale data sets, and at the same time meets different user needs based on multiple data versions in the data version list, so that any multiple versions of data can be scheduled and processed at the same time, ensuring the diversity of production data and rapid iterative updates.

[0056] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and no further description will be given later. On this basis, the first data production task includes target data version information and target original data information, and the first data production task type is determined by the data production demand. The first data production task type includes the first type and the second type. Please refer to Figure 2 , the step 120 includes steps S210 to S230: Step S210, when the first data production task type is the first type, determining target data version information according to data production requirements and a preset data version list, and determining target original data according to data production requirements and a preset scheduling algorithm, to obtain target original data information; In this embodiment, the first data production task type is set for different selection methods of target original data. The first type refers to the data production task type generated based on the original data within the time selected by the user in the data production requirement; the second type refers to the data production task type generated based on the specified original data selected by the user in the data production requirement.

[0057] Specifically, when the first data production task type is the first type, that is, the user selects the original data corresponding to the first data production task according to the time point on the data production requirement input page, the data management platform will first determine the target data version information associated with the first data production task based on the data production requirement and the preset data version list.

[0058] Then, based on the time range selected by the user in the data production requirements, the preset scheduling algorithm is used to search for the original data within the corresponding time range in the database of the data management platform, and it is used as the target original data to obtain the target original data information.

[0059] For example, Figure 6 As shown, on the data production demand input page, the user selects the type of mining task (i.e., data production task) as time point, and determines the data range of the original data associated with the data production task by setting the start time and end time.

[0060] Step S220: when the first data production task type is the second type, determining target data version information and target original data information according to data production requirements and a preset data version list; Specifically, when the first data production task type is the second type, that is, the user directly specifies the original data corresponding to the first data production task on the input page of the data production requirements, the data management platform will determine the target data version information and target original data information associated with the first data production task based on the original data specified by the data production requirements, the specified data version and the preset data version list.

[0061] In this embodiment, when the first data production task type is the second type, the user selects the first data production task type as specified data on the input page of the data production requirements. The preset scheduling algorithm will display all the original data information in the database of the data management platform. The user can filter and select the original data by setting different dimensions (such as data type, vehicle model, vehicle number and related routes of the data source, etc.), and the user is supported to select any multiple data as specified original data.

[0062] Step S230: creating a first data production task based on the target data version information and the target original data information.

[0063] Finally, the first data production task is created based on the target data version information and the target original data information in combination with the task priority determined by the data production requirements.

[0064] The present embodiment provides a data production method, which receives a data production demand from a user; when the first data production task type is of the first type, the target data version information is determined according to the data production demand and a preset data version list, and the target original data is determined according to the data production demand and a preset scheduling algorithm to obtain the target original data information; when the first data production task type is of the second type, the target data version information and the target original data information are determined according to the data production demand and the preset data version list; based on the target data version information and the target original data information, a first data production task is created; based on the first data production task, a data scheduling queue is created; based on the machine cluster, the preset scheduling algorithm and the data scheduling queue, data production is performed to obtain a data production result.

[0065] Through the above scheme, this embodiment uses two types of data production tasks and combines them with a preset data version list to create a data production task, then creates a data scheduling queue based on the data production task, and then combines the machine cluster and the preset scheduling algorithm to produce data. At the same time, based on multiple data versions in the data version list and multiple data production task types determined by user needs, different user needs can be more comprehensively met, and any multiple versions of data can be scheduled and processed at the same time, ensuring the diversity of production data and rapid iterative updates.

[0066] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment and the second embodiment can refer to the above introduction, and will not be repeated later. Figure 3 , before the above step S110, it also includes steps S310 to S340: Step S310, receiving the user's data version requirement; Specifically, the data management platform first needs to receive the user's data version requirements and build the data version according to the existing data usage requirements and data analysis model information. The data management platform usually receives the user's data version requirements through multiple form components in a visual page.

[0067] For example, refer to Figure 7 , the data management platform uses Figure 7 The page shown receives the user's data version request.

[0068] Step S320, determining version information of the data version based on the data version requirement; Then, based on the received data version requirements, the version information of the data version required by the user is obtained, including the name, organization, version type, version characteristics and associated branching algorithm.

[0069] The name is the data version name, which is used to distinguish different data versions; the version attribute is used to distinguish whether the created data version is the default version. The organization, version type and associated branch algorithm have been explained in the previous embodiment, and will not be repeated here.

[0070] Step S330, creating a data version according to the version information; Step S340: Generate the data version list based on the data version.

[0071] It should be understood that data versions are created based on the above version information, and multiple data versions can be constructed based on the data version requirements of several users, thereby generating the data version list to meet the diverse data production needs of users and support multiple versions of data to be scheduled for processing.

[0072] This embodiment provides a data production method, which includes receiving a data version requirement from a user; determining version information of a data version based on the data version requirement; creating a data version according to the version information; generating a data version list based on the data version; receiving a data production requirement from a user; creating a first data production task based on the data production requirement and a preset data version list; creating a data scheduling queue based on the first data production task; and performing data production based on the machine cluster, a preset scheduling algorithm, and a data scheduling queue to obtain a data production result.

[0073] Through the above scheme, this embodiment receives the user's data version requirements, obtains the version information that the user needs to create, and constructs the data version based on the version information, thereby constructing a preset data version list, which is convenient for subsequent simultaneous scheduling and production of multiple versions of data based on the preset data version list combined with the user's data production requirements, so as to meet the user's diverse data production requirements.

[0074] Based on the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the third embodiment can be referred to the above introduction, and no further description will be given later. On this basis, the version information includes version characteristics, please refer to Figure 4 , after the above step 340, further comprising steps S410 to S40: Step S410: if the version characteristic of the data version is a preset characteristic, automatically creating a second data production task based on the data version; Step S420, based on the second data production task, confirming the target original data and the task priority; Step S430, based on the target original data and the task priority, a data scheduling queue is created, and the step of executing: based on the machine cluster, the preset scheduling algorithm and the data scheduling queue, data production is performed to obtain a data production result; In this embodiment, the preset characteristic is the default version, and the version characteristic is used to indicate whether the data version is the default version.

[0075] Specifically, after creating a data version based on the user's data version requirements, the version characteristics of the data version are automatically detected to see if they are the default version. For the default version of the data version, the production of its scheduling data is automatically triggered by a preset scheduling algorithm.

[0076] When the version characteristic of the created data version is the default version, the data management platform can automatically create a second data production task directly based on the data version. It should be noted that the second data production task does not include the target original data information but only includes the target data version information.

[0077] Then, according to the preset scheduling algorithm, the database of the data management platform is automatically monitored for updates of the original data. Once the original data that has been updated and preliminarily parsed is detected, the original data is used as the target original data of the second data production task. Then, the priority of task processing can be obtained according to the processing volume of the target original data, the update time of the target original data, the target data version and other information. Finally, the data scheduling queue is created according to the target original data carrying the task priority and the target data version information. Finally, step 140 is executed again: based on the machine cluster, the preset scheduling algorithm and the data scheduling queue, data production is performed to obtain the data production result, thereby producing the data corresponding to the requirements of the first data production task.

[0078] It should be understood that even after the second data production task is automatically created based on the preset characteristics of the version characteristics of the data version, the user's data production needs can still be received, thereby creating the first data production task and producing data. Since the task of the first data production task is determined based on user needs, more diverse data can be produced.

[0079] Step S440: If the version characteristic of the data version is not a preset characteristic, the step of receiving the data production demand of the user is executed.

[0080] When the version characteristic of the created data version is a non-default version, the data management platform needs to manually specify the original data information of data production in order to create a data production task and thereby create a data scheduling queue. That is, it is necessary to execute step S110: receiving the user's data production requirements.

[0081] The present embodiment provides a data production method, which receives a data version requirement from a user; determines version information of a data version based on the data version requirement; creates a data version according to the version information; generates a data version list based on the data version; if the version characteristic of the data version is a preset characteristic, automatically creates a second data production task based on the data version; based on the second data production task, confirms target original data and task priority; creates a data scheduling queue based on the target original data and task priority, and executes the steps of: performing data production based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain a data production result; if the version characteristic of the data version is not a preset characteristic, executing the steps of: receiving a data production requirement from a user; creating a first data production task based on the data production requirement and the preset data version list; creating a data scheduling queue based on the first data production task; performing data production based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain a data production result.

[0082] Through the above scheme, this embodiment confirms that the version characteristic of the data version is the default version, thereby realizing the scheduling of automated data production for the latest original data of the data management platform without manual operation, so that multiple data versions can be set as the default version, and the production of multiple versions of data can be simultaneously scheduled to meet the diverse data production needs of users.

[0083] Based on the first embodiment and / or the second embodiment and / or the third embodiment and / or the fourth embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the first embodiment, the second embodiment, the third embodiment and / or the fourth embodiment can refer to the above description and will not be described in detail later. Figure 5 , after the above step 140, including steps S510~S530: Step S510, searching and managing the data version according to a preset data version list; Step S520, managing the data production task according to the data scheduling queue; Specifically, according to the aforementioned data version list, not only can data versions be retrieved by data version name, organization, version type, version characteristics, and associated branching algorithm, but multiple data versions can also be edited, modified, and deleted. Figure 8 , users can use the data management platform to batch set the version characteristics of multiple data versions as the default version.

[0084] The data management platform can also manage the processing priority of data production tasks based on the previously created data scheduling queue, and support one-time batch modification of multiple data production tasks.

[0085] Step S530, uploading and storing the data production results to the cloud database of the data management platform, and deleting the intermediate output data on the machine cluster.

[0086] It is understandable that after the machine cluster performs relevant data production according to the data production task, in order to improve the utilization rate of the machine cluster, the intermediate output data occupying the machine cluster resources can be deleted after the production data is stored.

[0087] Specifically, the production data, the original data source of the production data, the associated data version, the associated team and other information are uploaded and stored in the cloud database of the data management platform, and the intermediate output data on the machine cluster is deleted.

[0088] This embodiment provides a data production method, which receives a user's data production demand; creates a first data production task based on the data production demand and a preset data version list; creates a data scheduling queue based on the first data production task; performs data production based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain a data production result; retrieves and manages the data version according to the preset data version list; and manages the data version according to the data scheduling queue. The data production result is uploaded and stored in the cloud database of the data management platform, and the intermediate output data on the machine cluster is deleted.

[0089] This embodiment provides a data production method. Through a preset data version list and a data scheduling queue, users can manage data versions and data production tasks. By saving data production results to the cloud and deleting intermediate output data on the machine cluster, it supports dynamic management of data versions and data scheduling queues, while improving the processing performance of the machine cluster and further improving the efficiency of data production.

[0090] This application also provides a data production device, please refer to Fig. 9 , the data production device is applied to a data management platform, the data management platform includes a machine cluster, and the data production device includes: A receiving module 10 is used to receive a user's data production demand; A task creation module 20, used to create a first data production task based on the data production requirement and a preset data version list; A queue creation module 30, used to create a data scheduling queue based on the first data production task; The data production module 40 is used to produce data based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain data production results.

[0091] The data production device provided by the present application adopts the data production method in the above embodiment, which can solve the technical problem that the existing data production method cannot quickly iterate and update when producing data for large data sets. Compared with the prior art, the beneficial effects of the data production device provided by the present application are the same as the beneficial effects of the data production method provided by the above embodiment, and the other technical features in the data production device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0092] The present application provides a data production device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data production method in the above-mentioned embodiment one.

[0093] Reference below Fig.10 , which shows a schematic diagram of the structure of a data production device suitable for implementing the embodiment of the present application. The data production device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The data production device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0094] like Fig.10As shown, the data production device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data production device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data production device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data production device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0095] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0096] The data production device provided by the present application adopts the data production method in the above embodiment, which can solve the technical problem that the existing data production method cannot quickly iterate and update when producing data for large data sets. Compared with the prior art, the beneficial effects of the data production device provided by the present application are the same as the beneficial effects of the data production method provided by the above embodiment, and the other technical features in the data production device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0097] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0098] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0099] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data production method in the above-mentioned embodiment.

[0100] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0101] The above-mentioned computer-readable storage medium may be included in the data production device; or it may exist independently without being assembled into the data production device.

[0102] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data production device, the data production device: receives the user's data production requirements; creates a first data production task based on the data production requirements and a preset data version list; creates a data scheduling queue based on the first data production task; performs data production based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain data production results.

[0103] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0106] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data production method, and can solve the technical problem that the existing data production method cannot quickly iterate and update when producing data for large data sets. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data production method provided in the above-mentioned embodiment, and will not be repeated here.

[0107] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned data production method when executed by a processor.

[0108] The computer program product provided by the present application can solve the technical problem that the existing data production method cannot quickly iterate and update when producing data for large data sets. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the data production method provided by the above embodiment, and will not be repeated here.

[0109] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A data production method, characterized in that: The method is applied to a data management platform, the data management platform includes a machine cluster, and the method includes: Receive data production requirements from users; Based on the data production requirement and the preset data version list, create a first data production task; Based on the first data production task, create a data scheduling queue; Based on the machine cluster, the preset scheduling algorithm and the data scheduling queue, data production is performed to obtain data production results.

2. The method according to claim 1, characterized in that The first data production task includes target data version information and target original data information. The first data production task type is determined by the data production requirement. The first data production task type includes a first type and a second type. The step of creating the first data production task based on the data production requirement and the preset data version list includes: When the first data production task type is the first type, determining target data version information according to data production requirements and a preset data version list, and determining target original data according to data production requirements and a preset scheduling algorithm to obtain target original data information; When the first data production task type is the second type, determining target data version information and target original data information according to data production requirements and a preset data version list; Based on the target data version information and the target original data information, a first data production task is created.

3. The method according to claim 2, characterized in that Before the step of receiving the user's data production demand, the method further includes: Receive data version requirements from users; Based on the data version requirement, determine version information of the data version; Creating a data version according to the version information; Based on the data version, the data version list is generated.

4. The method according to claim 3, characterized in that The version information includes version characteristics. After the step of generating the data version list based on the data version, the step further includes: If the version characteristic of the data version is a preset characteristic, automatically creating a second data production task based on the data version; Based on the second data production task, confirm the target original data and task priority; Based on the target original data and the task priority, a data scheduling queue is created, and the following steps are performed: based on the machine cluster, the preset scheduling algorithm and the data scheduling queue, data production is performed to obtain a data production result; If the version characteristic of the data version is not a preset characteristic, the step of receiving a user's data production demand is executed.

5. The method according to claim 2, characterized in that The step of performing data production based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain the data production result includes: Using a preset scheduling algorithm, regularly detecting the resource status of the machine cluster according to a preset period; According to the resource status, scheduling target original data from the data scheduling queue to obtain target data information; Sending the target data information to the machine cluster; Based on the machine cluster, data production is performed according to the target data information to obtain a data production result.

6. The method according to claim 5, characterized in that The target data information includes the target data version information, and the target data version information includes an associated branch algorithm. The step of performing data production based on the machine cluster and the target data information to obtain a data production result includes: Based on the machine cluster, configure and deploy containers according to the target data information; Based on the container, target data is acquired, and data production is performed on the target data according to the association branch algorithm to obtain a data production result.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Retrieving and managing the data versions according to a preset data version list; Managing the data production tasks according to the data scheduling queue; The data production results are uploaded and stored in the cloud database of the data management platform, and the intermediate output data on the machine cluster is deleted.

8. A data production device, characterized in that: The data production device is applied to a data management platform, the data management platform includes a machine cluster, and the data production device includes: A receiving module, used to receive data production requirements from users; A task creation module, used to create a first data production task based on the data production requirement and a preset data version list; A queue creation module, used to create a data scheduling queue based on the first data production task; The data production module is used to produce data based on the machine cluster, the preset scheduling algorithm and the data scheduling queue to obtain data production results.

9. A data production device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data production method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data production method according to any one of claims 1 to 7 are implemented.

Images