A data reasoning method, device, equipment and storage medium

Abstract

The application discloses a data reasoning method and device, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: obtaining offline data to be reasoned; creating a plurality of reasoning tasks for the offline data, and respectively configuring resources occupied by the reasoning tasks to obtain corresponding resource configuration information; taking the offline data as input and executing each reasoning task according to the resource configuration information, so as to perform reasoning analysis on the offline data through the reasoning task and obtain corresponding reasoning results. According to the application, the resources occupied by each reasoning task are configured when the reasoning task is created, and the reasoning task can be automatically scheduled according to the resource configuration information, so that automatic reasoning analysis of offline data is realized, the reasoning process of offline data is simplified, and the labor cost is saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data reasoning method and device, equipment and a storage medium. BACKGROUND

[0002] At present, when performing reasoning analysis on offline data, the scheduling of a reasoning task created by a Job (a task type provided by k8s) controller is usually performed in an artificial manner, such as by a developer to perform the scheduling of the reasoning task, and then the corresponding data reasoning result is obtained by executing the reasoning task. However, due to the short execution time and large quantity of reasoning tasks, there are many difficulties in manual scheduling and maintenance. In addition, a large amount of development work is required for data analysis after the execution of the reasoning task, such as storage and analysis of abnormal data, investigation of the failure reason of the reasoning task, and the like. In addition, in the process of performing reasoning analysis on offline data, there may be a case that the reasoning accuracy of a trained model for a certain type of picture or sentence data is very poor, and the specific reason for the failure of the reasoning task cannot be obtained after the failure of the reasoning task. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a data reasoning method, device, equipment and storage medium, which can automatically schedule reasoning tasks, realize automatic reasoning analysis of offline data, simplify the reasoning process of offline data, and save labor costs. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a data reasoning method, comprising:

[0005] obtaining offline data to be reasoned;

[0006] creating a plurality of reasoning tasks for the offline data, and respectively configuring resources occupied by the reasoning tasks to obtain corresponding resource configuration information;

[0007] executing each of the reasoning tasks according to the resource configuration information with the offline data as input, so as to perform reasoning analysis on the offline data by the reasoning tasks to obtain corresponding reasoning results.

[0008] Optionally, after the respective configuration of the resources occupied by the reasoning tasks to obtain the corresponding resource configuration information, the method further comprises:

[0009] respectively setting any one or more of the following parameters for the reasoning tasks: a predicted running duration, a running level and a timing running time;

[0010] Correspondingly, the execution of each of the reasoning tasks according to the resource configuration information comprises:

[0011] According to the resource configuration information and any one or more of the set expected running time length, the running level and the timing running time, each of the inference tasks is executed.

[0012] Optionally, the execution of each of the inference tasks according to the resource configuration information and any one or more of the set expected running time length, the running level and the timing running time comprises:

[0013] If the inference task is set with the timing running time, it is determined whether the current running time reaches the timing running time;

[0014] If the current running time reaches the timing running time, the inference task reaching the timing running time is directly executed;

[0015] If the current running time does not reach the timing running time or the inference task is not set with the timing running time, it is determined whether the inference task is set with the running level, and if the inference task is set with the running level, the inference tasks are executed in the order from high to low according to the running level;

[0016] If the inference task is not set with the running level or the running levels are the same, the inference tasks are executed in the order from small to large according to the resource configuration in the resource configuration information;

[0017] If the resource configurations in the resource configuration information are the same, the inference tasks are executed in the order from short to long according to the expected running time length.

[0018] Optionally, after the inference analysis on the offline data by the inference tasks is performed to obtain corresponding inference results, the method further comprises:

[0019] All the inference tasks are deleted.

[0020] Optionally, in the process of creating the inference tasks, the method further comprises:

[0021] The data access requests for the current system are captured by adding a logger keyword, and each of the data access requests is parsed to obtain parsed data;

[0022] The parsed data is forwarded to an inference data collection interface, so that the parsed data is calculated by the inference data collection interface to obtain target attribute information corresponding to a preset HTTP attribute.

[0023] Optionally, the data inference method further comprises:

[0024] When the data access request and / or the data contained in the target attribute information are file data, the distributed file system or local storage is used as external storage to store the data in the data access request and / or the data in the target attribute information, and the mapping relationship between the main storage Elasticsearch and the distributed file system or the local storage is recorded in the main storage Elasticsearch.

[0025] Optionally, the data inference method further includes:

[0026] Export the execution logs within the inference task in file format.

[0027] Secondly, this application discloses a data inference apparatus, comprising:

[0028] The data acquisition module is used to acquire offline data to be inferred.

[0029] The task creation module is used to create multiple inference tasks for the offline data;

[0030] The resource configuration module is used to configure the resources occupied by the inference task respectively, and obtain the corresponding resource configuration information;

[0031] The task execution module is used to take the offline data as input and execute each of the inference tasks according to the resource configuration information, so as to perform inference analysis on the offline data through the inference tasks and obtain the corresponding inference results.

[0032] Thirdly, this application discloses an electronic device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the aforementioned data reasoning method.

[0033] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned data reasoning method.

[0034] As can be seen, this application first acquires the offline data to be inferred, then creates multiple inference tasks for the offline data, and configures the resources occupied by each inference task to obtain corresponding resource configuration information. The offline data is then used as input, and each inference task is executed according to the resource configuration information, so that the offline data can be analyzed through the inference tasks to obtain corresponding inference results. This application, by configuring the resources occupied by each inference task during creation and automatically scheduling the inference tasks according to the resource configuration information, achieves automatic inference analysis of offline data, simplifies the inference process of offline data, and saves labor costs. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0036] Figure 1 A data reasoning method flow chart disclosed by the present application;

[0037] Figure 2 A resource configuration schematic diagram of a specific Job creating reasoning task disclosed by the present application;

[0038] Figure 3 A specific data analysis method schematic diagram disclosed by the present application;

[0039] Figure 4 A configuration file and configuration parameter meaning corresponding relationship schematic diagram disclosed by the present application;

[0040] Figure 5 A specific reasoning data batch analysis method schematic diagram disclosed by the present application;

[0041] Figure 6 A specific reasoning data collection method schematic diagram disclosed by the present application;

[0042] Figure 7 A specific data reasoning method flow chart disclosed by the present application;

[0043] Figure 8 A specific reasoning task scheduling method flow chart disclosed by the present application;

[0044] Figure 9 A data reasoning device structure schematic diagram disclosed by the present application;

[0045] Figure 10 A specific data reasoning device structure schematic diagram disclosed by the present application; DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] The embodiments of the present application disclose a data reasoning method, as shown inFigure 1 As shown, the method comprises:

[0048] Step S11: obtaining offline data to be inferred.

[0049] In this embodiment, first, historical offline data generated by a service end and to be subjected to an inference task is obtained, wherein the format of the offline data includes but is not limited to a text format, a video format, a picture format, and the like. In a specific implementation manner, the historical offline data can be specifically video stream monitoring data to be subjected to a number of off-duty statistics task.

[0050] Step S12: creating a plurality of inference tasks for the offline data, and respectively configuring resources occupied by the inference tasks to obtain corresponding resource configuration information.

[0051] In this embodiment, after the offline data to be inferred is obtained, a Job controller can be used to create a plurality of inference tasks for the offline data, wherein the Job controller is composed of one or more Pods, and a Pod is composed of one or more containers. It should be noted that the inference tasks can be different inference tasks created for the same group of offline data, such as inference tasks of total number statistics and target personnel searching created for the same group of offline video stream data, or can be different inference tasks of total number statistics respectively created for the offline data including video stream and pictures. Further, after the plurality of inference tasks are created for the offline data, resources occupied by the inference tasks are respectively configured to obtain corresponding resource configuration information. The resources occupied by the inference tasks include but are not limited to a CPU (Central Processing Unit), a memory, an acceleration card, and the like, and the resource configuration can determine the size of resources occupied by the inference tasks during running. Referring to Figure 2 As shown, Figure 2 a simple resource configuration manner of creating inference tasks using a Job is shown.

[0052] Step S13: inputting the offline data as input and executing each of the inference tasks according to the resource configuration information, so as to perform inference analysis on the offline data through the inference tasks to obtain corresponding inference results.

[0053] In this embodiment, after the resources occupied by the inference tasks are respectively configured to obtain corresponding resource configuration information, the offline data can be input into each of the inference tasks, then the configuration resources in the resource configuration information are sorted, and each of the inference tasks is executed in the order of the configuration resources from small to large. Through the execution of the inference tasks, inference analysis on the offline data can be implemented, and further corresponding inference results can be obtained.

[0054] Further, if the user wants to further parse the inference result and the inference result is of a text type, the user can be provided with a standardized parsing template for the positioning problem of the inference task. The specific implementation process can be to fill in the parameters according to the standardized parsing template, and then perform data parsing according to the filled parameters. For details, refer to Figure 3 , Figure 3 the specific meanings of the configuration parameters in Figure 4 are shown. In addition, refer to Figure 5 , the inference result can also be batch parsed, and batch parsing can be achieved by writing the configurations of two inference tasks in a configuration file at one time.

[0055] It should be pointed out that in the process of creating the inference task, it further includes: grabbing the data access request for the current system by adding the logger keyword, and parsing each of the data access requests to obtain parsed data; forwarding the parsed data to the inference data collection interface, so as to perform parsing calculation on the parsed data through the inference data collection interface to obtain target attribute information corresponding to a preset HTTP attribute. In this embodiment, in order to echo the data of the online inference task of the system, in the process of creating the inference task for offline data, the data echo of the current inference task can also be achieved by adding the logger keyword in the Job controller. The specific implementation process includes: when creating the inference task for offline data, a default container default-proxy is set, so that each data access request for the current system (i.e. various HTTP (Hyper Text Transfer Protocol, Hyper Text Transfer Protocol) requests) passes through the above default container default-proxy, then the data access request passing through the above default container default-proxy is grabbed, then each of the grabbed data access requests is parsed to obtain parsed data including various HTTP attributes, and then the parsed data is forwarded to a pre-created inference data collection interface, such as a URL (Uniform Resource Locator, Uniform Resource Locator). For details, refer to Figure 6As shown, when data echoing needs to be performed, it can be implemented by adding a logger keyword under the spec->template->spec level of the configuration file, and configuring the inference data collection interface URL for the logger. Further, after the inference data collection interface obtains the parsed data forwarded by the logger, it will perform parsing calculation on the parsed data, and then obtain target attribute information corresponding to the preset HTTP attribute. The specific calculated target HTTP attribute information includes but is not limited to request time, response time, processing time, request data type, request data metadata, response data type, response data metadata, response state and other key information. Through the above target HTTP attribute information, the HTTP attribute can be restored, and other new HTTP attributes such as request id can also be added according to the requirements, and then the routing relationship is configured through the URL keyword, and is forwarded to the inference data collection interface.

[0056] Further, when the data contained in the data access request and / or the target attribute information is file data, a distributed file system (HDFS, Hadoop Distributed File System) or a local storage can be used as external storage to store the data in the data access request and / or the target attribute information, and then a mapping relationship between the main storage Elasticsearch (a search server based on Lucene) and the distributed file system or the local storage is recorded in the main storage Elasticsearch. In this embodiment, after the target attribute information corresponding to the preset HTTP attribute is obtained by parsing and calculating the parsed data through the inference data collection interface, the target attribute information needs to be stored. Considering that the input and output types of different inference services are different, there are various file types of attribute information such as pictures, texts, and videos, so two storages can be used to store them respectively. In a first specific implementation, Elasticsearch is used as the main storage, and HDFS is used as the external file storage. When the data in the data access request and the target attribute information is of a file type, the mapping relationship between the main storage Elasticsearch and HDFS can be stored, and the specific file data, i.e., the data in the data access request and the target attribute information, is stored in HDFS. The above storage method is used because Elasticsearch has superior performance in text storage and query, but is not suitable for storing files, and HDFS, as a mainstream distributed storage system, can solve the shortcomings of Elasticsearch. In another specific implementation, Elasticsearch can be used as the main storage, and a local storage can be used as the external file storage. For example, a local disk path is mounted to the inference task through the volume of k8s. In this way, if the stored inference data is mostly small files, the performance bottleneck problem of a large number of small files in query when HDFS is used as the external file storage can be solved. Of course, the above two storage methods can be combined.

[0057] Further, after the inference analysis of the offline data is performed through the inference task to obtain the corresponding inference result, all the inference tasks can be deleted. It can be understood that the inference task generally has a limited execution time and is different from the online inference task with a long execution time. Therefore, the inference task should be released as soon as possible after the inference analysis is performed, so as to arrange the execution of subsequent inference tasks. In this embodiment, when the inference task is executed, the configured resources need to be released, i.e., all the executed inference tasks are deleted, regardless of whether the inference task is executed successfully or fails. Of course, the inference task can also be deleted manually by a user.

[0058] Further, after the inference task is scheduled to be executed, the execution log inside the inference task can be further exported in a file format. In this embodiment, the execution log recorded inside the inference task is exported regardless of whether the inference task is successfully executed or not. The specific export format can be a file format. By exporting the execution log, the inference task that fails to be executed can be quickly located and the specific failure reason can be analyzed. In a specific implementation, when the inference task is created, the execution log recorded inside the inference task can be exported according to the option of starting all logs, starting success logs or starting failure logs selected by the user.

[0059] It can be seen that, in the embodiment of the application, the offline data to be inferred is first acquired, then a plurality of inference tasks are created for the offline data, the resources occupied by the inference tasks are respectively configured to obtain corresponding resource configuration information, the offline data is taken as input and each inference task is executed according to the resource configuration information, so that the inference analysis on the offline data is performed by the inference tasks to obtain corresponding inference results. In the embodiment of the application, the resources occupied by each inference task are configured when the inference task is created, and the inference task can be automatically scheduled according to the resource configuration information, so that the automatic inference analysis of the offline data is realized, the inference process of the offline data is simplified, and the labor cost is saved.

[0060] The embodiment of the application discloses a specific data inference method, as shown in Figure 7 The method comprises the following steps:

[0061] Step S21: acquiring offline data to be inferred.

[0062] Step S22: creating a plurality of inference tasks for the offline data, and respectively configuring the resources occupied by the inference tasks to obtain corresponding resource configuration information.

[0063] Step S23: setting any one or more parameters of a predicted running duration, a running level and a timing running time for each inference task.

[0064] In this embodiment, after the resources occupied by the inference tasks are respectively configured to obtain the corresponding resource configuration information, other parameters of the inference tasks can be further set, specifically including any one or more parameters of a predicted running duration, a running level and a timing running time. The running level includes but is not limited to emergency, high, medium, low, etc.

[0065] Step S24: inputting the offline data, and executing each of the inference tasks according to the resource configuration information and any one or more of the set expected running time length, the running level, and the timing running time, so as to perform inference analysis on the offline data by the inference tasks to obtain corresponding inference results.

[0066] In this embodiment, after any one or more of the expected running time length, the running level, and the timing running time is set for the inference task, the offline data can be inputted, and each of the inference tasks can be executed according to the resource configuration information and any one or more of the set expected running time length, the running level, and the timing running time, so as to perform inference analysis on the offline data by the inference tasks to obtain corresponding inference results.

[0067] Specifically, referring to FIG. 4, the execution of each of the inference tasks according to the resource configuration information and any one or more of the set expected running time length, the running level, and the timing running time can include: Figure 8

[0068] Step S31: if the timing running time is set for the inference task, determining whether the current running time reaches the timing running time;

[0069] Step S32: if the current running time reaches the timing running time, directly running the inference task reaching the timing running time;

[0070] Step S33: if the current running time does not reach the timing running time or the inference task does not set the timing running time, determining whether the inference task sets the running level, and if the running level is set, executing the inference tasks in the order from high to low according to the running level;

[0071] Step S34: if the inference task does not set the running level or the running levels are the same, executing the inference tasks in the order from small to large according to the resource configuration in the resource configuration information;

[0072] Step S35: if the resource configuration in the resource configuration information is the same, executing the inference tasks in the order from short to long according to the expected running time length.

[0073] ​For example, when the three parameters of the expected running time, the running level and the timing running time are set for each reasoning task respectively, firstly, it is judged whether the current running time has reached the preset timing running time, if there is a reasoning task whose current running time has reached the timing running time, the reasoning task is directly run, that is, the timing running time has the highest priority; if there is no reasoning task whose current running time has reached the timing running time, then each reasoning task is executed in the order from high to low according to the set running level, for example, each reasoning task is executed according to the running level of emergency, high, medium and low; if there are reasoning tasks with the same running level, then the size of the resource configuration in each resource configuration information is further judged, and then each reasoning task is executed in the order from small to large according to the resource configuration; if the size of the resource configuration in the resource configuration information is also the same, then each reasoning task is executed in the order from short to long according to the set expected running time.

[0074] The more specific processing procedures of steps S21 and S22 will be described below with reference to the corresponding contents disclosed in the foregoing embodiments.

[0075] It can be seen that, in the embodiment, the offline data to be reasoned is first acquired, then multiple reasoning tasks are created for the offline data, and the resources occupied by the reasoning tasks are configured respectively to obtain corresponding resource configuration information, then any one or more of the parameters of the expected running time, the running level and the timing running time is set for each reasoning task, the offline data is taken as input, and each reasoning task is executed according to the resource configuration information and any one or more of the set parameters of the expected running time, the running level and the timing running time, so as to perform reasoning analysis on the offline data through the reasoning tasks to obtain a corresponding reasoning result. In the embodiment, when the reasoning tasks are created, the resources occupied by each reasoning task and other preset parameters are configured, and through the formulated reasoning task scheduling rule, the automatic scheduling of multiple reasoning tasks can be realized, and then the automatic reasoning analysis of offline data is realized, so that the reasoning process of offline data is simplified, and the labor cost is saved.

[0076] Correspondingly, the embodiment of the application also discloses a data reasoning device, as shown in Figure 9 The device comprises:

[0077] The data acquisition module 11 is configured to acquire offline data to be reasoned.

[0078] The task creation module 12 is configured to create multiple reasoning tasks for the offline data.

[0079] The resource configuration module 13 is configured to configure resources occupied by the inference tasks respectively to obtain corresponding resource configuration information.

[0080] The task execution module 14 is configured to execute each of the inference tasks according to the resource configuration information with the offline data as input, so as to perform inference analysis on the offline data by the inference tasks to obtain corresponding inference results.

[0081] The specific workflow of each of the above modules can refer to the corresponding content disclosed in the foregoing embodiments, which will not be described here in detail.

[0082] It can be seen that in the embodiments of the present application, the offline data to be inferred is first obtained, then a plurality of inference tasks are created for the offline data, resources occupied by the inference tasks are configured respectively to obtain corresponding resource configuration information, and then the offline data is input and each of the inference tasks is executed according to the resource configuration information, so as to perform inference analysis on the offline data by the inference tasks to obtain corresponding inference results. In the embodiments of the present application, the resources occupied by each of the inference tasks are configured when the inference tasks are created, and the inference tasks can be automatically scheduled according to the resource configuration information, thereby realizing automatic inference analysis of the offline data, simplifying the inference process of the offline data, and saving labor cost.

[0083] In some specific embodiments, the resource configuration module 13 can further include:

[0084] a unit configured to set any one or more of a predicted running time, a running level, and a timing running time for each of the inference tasks;

[0085] Correspondingly, the task execution module 14 can specifically include:

[0086] a first task execution unit configured to execute each of the inference tasks according to the resource configuration information and any one or more of the predicted running time, the running level, and the timing running time.

[0087] In some specific embodiments, the first task execution unit can specifically include:

[0088] a first judgment unit configured to judge whether the current running time reaches the timing running time if the inference task is set with the timing running time;

[0089] a second task execution unit configured to directly run the inference task reaching the timing running time if the current running time reaches the timing running time.

[0090] The second judging unit is configured to judge whether the inference task is set with the running level if the current running time does not reach the timing running time or the inference task is not set with the timing running time.

[0091] The third task execution unit is configured to execute the inference task according to the running level from high to low if the running level is set.

[0092] The fourth task execution unit is configured to execute the inference task according to the resource configuration from small to large in the resource configuration information if the inference task is not set with the running level or the running level is the same.

[0093] The fifth task execution unit is configured to execute the inference task according to the predicted running time from short to long if the resource configuration size in the resource configuration information is the same.

[0094] In some embodiments, the task execution module 14 can further include:

[0095] The inference task deletion unit is configured to delete all the inference tasks.

[0096] In some embodiments, in the process of creating the inference task, the process can further include:

[0097] The access request grabbing unit is configured to grab the data access request for the current system by adding a logger keyword.

[0098] The request parsing unit is configured to parse each data access request to obtain parsed data.

[0099] The parsed data forwarding unit is configured to forward the parsed data to an inference data collection interface, so as to perform parsing calculation on the parsed data through the inference data collection interface to obtain target attribute information corresponding to a preset HTTP attribute.

[0100] In some embodiments, the data inference apparatus can further include:

[0101] The data storage unit is configured to store the data in the data access request and / or the target attribute information in a distributed file system or a local storage as external storage when the data in the data access request and / or the target attribute information is file data.

[0102] The mapping relationship recording unit is configured to record the mapping relationship between the main storage Elasticsearch and the distributed file system or the local storage in the main storage Elasticsearch.

[0103] In some specific embodiments, the data reasoning apparatus can further comprise:

[0104] a log exporting unit configured to export the execution log inside the reasoning task in a file format.

[0105] Further, the embodiments of the present application also disclose an electronic device, Figure 10 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the contents in the figure cannot be considered as any limitation on the use range of the present application.

[0106] Figure 10 A structural schematic diagram of an electronic device 20 provided by the embodiments of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, the computer program is loaded and executed by the processor 21 to implement the related steps in the data reasoning method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the present embodiment can be an electronic computer.

[0107] In the present embodiment, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input / output interface 25 is configured to obtain external input data or output data to the outside world, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0108] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0109] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the data reasoning method executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0110] Further, the application also discloses a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to realize the data inference method disclosed above. For the specific steps of the method, refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0111] The various embodiments are described in the specification by way of progression, each building on the last to facilitate ease of understanding. The same reference numerals are used in different embodiments and / or drawings to denote similar components. The embodiments disclosed in the specification are not meant to be limiting. Other embodiments can be used, and other changes can be made, without departing from the scope of the subject technology. Thus, the scope of the subject technology is indicated by the appended claims rather than by the description in the specification.

[0112] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be embodied in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples disclosed herein are described in terms of their functionality, their processing, and their output. Whether such functionality is implemented in hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0113] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0114] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but are used to identify one entity from another, and do not imply any actual relationship or sequence among entities. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0115] The data reasoning method, device, equipment and storage medium provided by the present application are described in detail above, and the principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A data reasoning method, characterized in that, include: Obtain the offline data to be inferred; Multiple inference tasks are created for the offline data, and the resources occupied by each inference task are configured to obtain the corresponding resource configuration information. The offline data is used as input, and each of the inference tasks is executed according to the resource configuration information, so as to perform inference analysis on the offline data through the inference tasks and obtain the corresponding inference results; The process of creating the inference task also includes: capturing data access requests for the current system by adding the keyword "logger", parsing each data access request to obtain parsed data; forwarding the parsed data to the inference data collection interface so that the parsed data can be parsed and calculated through the inference data collection interface to obtain target attribute information corresponding to preset HTTP attributes; the target attribute information includes request time, response time, processing time, request data type, request data metadata, response data type, response data metadata, and response status; The method of capturing data access requests for the current system by adding the keyword "logger" and parsing each data access request to obtain parsed data includes: setting a default container when creating an inference task for offline data so that each data access request for the current system passes through the default container, and capturing and parsing the data access requests that pass through the default container to obtain parsed data including various HTTP attributes.

2. The data reasoning method according to claim 1, characterized in that, After configuring the resources occupied by the inference task to obtain the corresponding resource configuration information, the process further includes: Set one or more parameters from the estimated runtime, runtime level, and timed runtime for the inference task; Accordingly, executing each of the inference tasks based on the resource configuration information includes: Each inference task is executed based on the resource configuration information and any one or more parameters set, including the estimated runtime, the runtime level, and the scheduled runtime.

3. The data reasoning method according to claim 2, characterized in that, The execution of each inference task based on the resource configuration information and any one or more parameters among the set estimated runtime, runtime level, and scheduled runtime includes: If the inference task has a set timed execution time, then determine whether the current execution time has reached the set timed execution time; If the current running time reaches the scheduled running time, then the inference task that has reached the scheduled running time will be run directly. If the current running time has not reached the scheduled running time or the inference task has not set the scheduled running time, then it is determined whether the inference task has set the running level. If the running level is set, the inference task is executed in descending order of the running level. If the inference task does not have the specified run level or the run levels are the same, the inference task is executed in ascending order of resource configuration in the resource configuration information. If the resource configuration sizes in the resource configuration information are the same, the inference tasks are executed in ascending order of the expected runtime.

4. The data reasoning method according to claim 1, characterized in that, After performing reasoning analysis on the offline data through the reasoning task to obtain the corresponding reasoning results, the method further includes: Delete all the aforementioned reasoning tasks.

5. The data reasoning method according to claim 1, characterized in that, Also includes: When the data access request and / or the data contained in the target attribute information are file data, the distributed file system or local storage is used as external storage to store the data in the data access request and / or the data in the target attribute information, and the mapping relationship between the main storage Elasticsearch and the distributed file system or the local storage is recorded in the main storage Elasticsearch.

6. The data reasoning method according to any one of claims 1 to 5, characterized in that, Also includes: Export the execution logs within the inference task in file format.

7. A data inference device, characterized in that, include: The data acquisition module is used to acquire offline data to be inferred. The task creation module is used to create multiple inference tasks for the offline data; The resource configuration module is used to configure the resources occupied by the inference task respectively, and obtain the corresponding resource configuration information; The task execution module is used to take the offline data as input and execute each of the inference tasks according to the resource configuration information, so as to perform inference analysis on the offline data through the inference tasks and obtain the corresponding inference results. The device is further configured to capture data access requests for the current system by adding the keyword "logger", and parse each data access request to obtain parsed data; and forward the parsed data to the inference data collection interface so that the parsed data can be parsed and calculated through the inference data collection interface to obtain target attribute information corresponding to preset HTTP attributes; The target attribute information includes request time, response time, processing time, request data type, request data metadata, response data data metadata, and response status; The device is also used to set a default container when creating an inference task for offline data, so that each data access request for the current system passes through the default container, and to capture and parse the data access requests that pass through the default container to obtain parsed data including various HTTP attributes.

8. An electronic device, characterized in that, It includes a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the data reasoning method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the data reasoning method as described in any one of claims 1 to 6.

Images