Big data code automatic detection method and device, equipment and medium

Abstract

The application relates to the technical field of big data, and provides a big data code automatic detection method and device, equipment and a medium, the method comprises the following steps: receiving big data code uploaded by a user, obtaining preset detection rule information, wherein the detection rule information comprises resource limitation occupied by total processes corresponding to the big data code, obtaining preset job process resource parameters in the big data code, detecting the preset job process resource parameters in the big data code, so as to determine whether the resource occupied by the total processes corresponding to the big data code exceeds the resource limitation, and generating a resource detection result corresponding to the big data code, and sending the resource detection result corresponding to the big data code to the user. According to the embodiment, the automatic detection of the big data code uploaded by the user is realized by obtaining the detection rule information, defects in the big data code can be detected, and the labor and resource costs are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a big data code automatic detection method and device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the further promotion of the enterprise Oracle strategy, some enterprises gradually migrate their business modules to big data systems. Due to the high difficulty and large workload of code testing in the big data environment, more stringent requirements are put forward for related big data codes. In addition, different departments and different projects have different requirements for big data codes. If manual detection is performed according to different detection requirements one by one before the big data codes are handed over, more manpower and resource costs will still be consumed. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a big data code automatic detection method, device, electronic device and computer readable storage medium, which can automatically scan and detect big data codes based on detection rules.

[0004] To achieve the above purpose, the first aspect of the embodiments of the present application provides a big data code automatic detection method, which comprises:

[0005] receiving a big data code uploaded by a user;

[0006] obtaining preset detection rule information, wherein the detection rule information comprises resource limits occupied by total processes corresponding to the big data code;

[0007] obtaining preset job process resource parameters in the big data code;

[0008] detecting the preset job process resource parameters in the big data code to determine whether the resources occupied by the total processes corresponding to the big data code exceed the resource limits, and generating a resource detection result corresponding to the big data code;

[0009] sending the resource detection result corresponding to the big data code to the user.

[0010] According to the big data code automatic detection method provided by some embodiments of the present application, the detection rule information further comprises data skew determination rules;

[0011] After obtaining the preset detection rule information, the method further comprises:

[0012] detecting whether the big data code has data skew based on the data skew determination rules, and generating a data skew detection result corresponding to the big data code;

[0013] The resource detection result corresponding to the big data code is sent to the user, including:

[0014] The resource detection result corresponding to the big data code and the data skew detection result are sent to the user.

[0015] According to the automatic detection method of the big data code provided by some embodiments of the present application, the data skew of the big data code is detected based on the data skew determination rule, and the data skew detection result corresponding to the big data code is generated, including:

[0016] The task execution information corresponding to the big data code is obtained through the resource management platform, and the task execution information includes the execution state of each node task and the processing data volume;

[0017] The average processing data volume is determined according to the processing data volume of each node task;

[0018] Based on the execution state of each node task and the size relationship between the processing data volume of each node task and the average processing data volume, the data skew detection result corresponding to the big data code is generated.

[0019] According to the automatic detection method of the big data code provided by some embodiments of the present application, the resource limit includes memory capacity limit, and the job process resource parameter includes job process quantity and content capacity occupied by each job process;

[0020] The preset job process resource parameter in the big data code is detected to determine whether the resource occupied by the total process corresponding to the big data code exceeds the resource limit, and the resource detection result corresponding to the big data code is generated, including:

[0021] The memory capacity occupied by the total process corresponding to the big data code is determined according to the job process quantity and the memory capacity occupied by each job process;

[0022] Based on the size relationship between the memory capacity occupied by the total process corresponding to the big data code and the memory capacity limit, the resource detection result corresponding to the big data code is generated.

[0023] According to the automatic detection method of the big data code provided by some embodiments of the present application, the resource limit includes processor core quantity limit, and the job process resource parameter includes job process quantity and processor core quantity occupied by each job process;

[0024] The preset job process resource parameter in the big data code is detected to determine whether the total process corresponding to the big data code occupies the resource exceeding the resource limit, and a resource detection result corresponding to the big data code is generated, comprising:

[0025] The number of processor cores occupied by the total process corresponding to the big data code is determined according to the number of job processes and the number of processor cores occupied by each job process;

[0026] The resource detection result of the big data code is generated based on the size relationship between the number of processor cores occupied by the total process corresponding to the big data code and the processor core number limit.

[0027] According to the automatic detection method of the big data code provided by some embodiments of the present application, after receiving the big data code uploaded by the user, the method further comprises:

[0028] The keyword detection is performed on the big data code to determine whether a preset target keyword exists in the big data code;

[0029] If it is determined that the preset target keyword exists in the big data code, the target keyword and the code statement containing the target keyword in the big data code are sent to the user.

[0030] According to the automatic detection method of the big data code provided by some embodiments of the present application, after receiving the big data code uploaded by the user, the method further comprises:

[0031] The syntax detection is performed on the big data code to determine whether a syntax error exists in the big data code;

[0032] If it is determined that the syntax error exists in the big data code, the code statement in which the syntax error exists in the big data code is sent to the user.

[0033] To achieve the above object, a second aspect of the embodiments of the present application provides an automatic detection device of big data code, comprising:

[0034] The receiving module is used for receiving the big data code uploaded by the user;

[0035] The first obtaining module is used for obtaining preset detection rule information, and the detection rule information comprises a resource limit occupied by a total process corresponding to the big data code;

[0036] The second obtaining module is used for obtaining a preset job process resource parameter in the big data code;

[0037] The detection module is configured to detect preset job process resource parameters in the big data code to determine whether resources occupied by total processes corresponding to the big data code exceed the resource limit, and to generate a resource detection result corresponding to the big data code.

[0038] The result sending module is configured to send the resource detection result corresponding to the big data code to the user.

[0039] To achieve the above object, a third aspect of embodiments of the present application provides an electronic device, which comprises a memory, a processor, a computer program stored in the memory and executable on the processor, and the computer program is executed by the processor to implement the method of the first aspect.

[0040] To achieve the above object, a fourth aspect of embodiments of the present application provides a storage medium, which is a computer readable storage medium, for computer readable storage, and the storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the method of the first aspect.

[0041] The present application provides an automatic detection method, device, electronic device and computer readable storage medium for big data code. The automatic detection method for big data code receives a big data code uploaded by a user and acquires preset detection rule information, the detection rule information comprising a resource limit occupied by total processes corresponding to the big data code. Then, preset job process resource parameters in the big data code are acquired, and the preset job process resource parameters in the big data code are detected to determine whether resources occupied by total processes corresponding to the big data code exceed the resource limit, and a resource detection result corresponding to the big data code is generated. Finally, the resource detection result corresponding to the big data code is sent to the user. By acquiring preset detection rule information, the present application realizes automatic detection of the big data code uploaded by the user, can detect defects in the big data code and reduce labor and resource costs. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of an automatic detection method for big data code provided by an embodiment of the present application;

[0043] Figure 2 is a flowchart of an automatic detection method for big data code provided by another embodiment of the present application;

[0044] Figure 3 is Figure 2 is a sub-step flowchart of step S210 in

[0045] Figure 4 is Figure 1 a sub-step flowchart of step S140 in

[0046] Figure 5 is Figure 1 a sub-step flowchart of step S140 in

[0047] Figure 6 is a flowchart of an automatic detection method of a big data code provided by another embodiment of the present application;

[0048] Figure 7 is a flowchart of an automatic detection method of a big data code provided by another embodiment of the present application;

[0049] Figure 8 is a structural schematic diagram of an automatic detection device of a big data code provided by an embodiment of the present application;

[0050] Figure 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0052] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0053] With the further promotion of the enterprise's Oracle strategy, some enterprises gradually migrate their business modules to big data systems. Due to the high difficulty and large workload of code testing in the big data environment, more stringent requirements are put forward for related big data codes. In addition, different departments and different projects have different requirements for big data codes. If manual detection is performed according to different detection requirements one by one before the handover of big data codes, it still needs to consume more manpower and resource costs.

[0054] Based on this, the embodiments of the present application provide an automatic detection method and device of a big data code, an electronic device and a computer readable storage medium, which can automatically scan and detect the big data code based on detection rules.

[0055] The application embodiment provides an automatic detection method and device of big data code, an electronic device and a computer readable storage medium. The automatic detection method of big data code in the application embodiment is described as follows.

[0056] The application embodiment can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain optimal results.

[0057] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes several major directions such as computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning.

[0058] The automatic detection method of big data code provided by the application embodiment can be applied to a terminal, can be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, big data and artificial intelligence platform, etc.; and the software can be an application for implementing the automatic detection method of big data code, but is not limited to the above forms.

[0059] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0060] See Figure 1 , Figure 1 A flowchart of an automatic detection method of a big data code provided by an embodiment of the application is shown. As shown in Figure 1 , the automatic detection method of the big data code includes but is not limited to steps S110 to S150:

[0061] Step S110, receiving a big data code uploaded by a user;

[0062] Step S120, obtaining preset detection rule information, the detection rule information including a resource limit occupied by total processes corresponding to the big data code;

[0063] Step S130, obtaining a preset job process resource parameter in the big data code;

[0064] Step S140, detecting the preset job process resource parameter in the big data code to determine whether the resource occupied by the total processes corresponding to the big data code exceeds the resource limit, and generating a resource detection result corresponding to the big data code;

[0065] Step S150, sending the resource detection result corresponding to the big data code to the user.

[0066] It can be understood that before formal handover of the big data code, the big data code uploaded by the user is obtained, and the preset detection rule information is obtained, and the big data code is detected according to the detection rule information to determine whether the big data code meets the corresponding rule requirements.

[0067] Further, the detection rule information includes a resource limit occupied by total processes corresponding to the big data code, the preset job process resource parameter in the big data code is acquired, then the preset job process resource parameter in the big data code is detected to determine whether the resource occupied by the total processes corresponding to the big data code exceeds the resource limit, a resource detection result corresponding to the big data code is generated, and finally the resource detection result corresponding to the big data code is sent to the user.

[0068] In one embodiment, the resource detection result includes the resource limit in the detection rule information, the preset job process resource parameter in the big data code, the resource occupied by the total processes corresponding to the big data code, and whether the resource exceeds the resource limit. The resource detection result containing the above information is sent to the user to inform the user of the job process resource parameter related to the resource limit in the uploaded big data code and whether the resource parameter setting exceeds the resource limit, thereby realizing resource warning of the big data code.

[0069] It can be understood that before performing automatic detection on the big data code, the detection rule information configured by the user can be acquired to realize the configurability of the rule.

[0070] It can be understood that in the case that different departments or different projects correspond to different detection rules, the user can be provided with the selection of multiple different detection rules, so that the big data code is scanned and detected according to the detection rule information selected by the user.

[0071] In another embodiment, the resource detection result corresponding to the big data code is sent to the user, including:

[0072] The email address of the user is acquired;

[0073] The resource detection result corresponding to the big data code is sent to the user through the email address.

[0074] In some embodiments, the resource limit includes a memory capacity limit, and the job process resource parameter includes a job process quantity and a content capacity occupied by each job process.

[0075] Referring to Figure 4 , Figure 4 is Figure 1 the sub-step flow chart of step S130 in Figure 4 As shown in the figure, the detection on the preset job process resource parameter in the big data code to determine whether the resource occupied by the total processes corresponding to the big data code exceeds the resource limit and generate the resource detection result corresponding to the big data code includes:

[0076] Step S410, determining the memory capacity occupied by the total processes corresponding to the big data code according to the number of job processes and the memory capacity occupied by each of the job processes;

[0077] Step S420, generating the resource detection result corresponding to the big data code based on the size relationship between the memory capacity occupied by the total processes corresponding to the big data code and the memory capacity limit.

[0078] For example, the job process resource parameters in the big data code are as follows:

[0079] num-executors: representing the number of Executor processes for executing Spark tasks;

[0080] executor-memory: representing the memory capacity occupied by each Executor process;

[0081] The performance of each Spark task depends largely on the content capacity of the Executor process, and the content capacity occupied by the total Executor processes corresponding to the big data code can be determined according to the number of Executor processes and the content capacity occupied by each Executor process.

[0082] It should be noted that the content capacity limit can be determined by the maximum content capacity in the department resource queue.

[0083] In some embodiments, the resource limit includes a processor core number limit, and the job process resource parameters include a job process and a processor core number occupied by each job process;

[0084] Referring to Figure 5 , Figure 5 is Figure 1 the sub-step flowchart of step S130 in Figure 5 As shown in the figure, the detection of the job process resource parameters preset in the big data code to determine whether the resources occupied by the total processes corresponding to the big data code exceed the resource limit, and to generate the resource detection result corresponding to the big data code, includes:

[0085] Step S510, determining the processor core number occupied by the total processes corresponding to the big data code according to the number of job processes and the processor core number occupied by each of the job processes;

[0086] Step S520, generating the resource detection result of the big data code based on the size relationship between the processor core number occupied by the total processes corresponding to the big data code and the processor core number limit.

[0087] Exemplarily, preset job process resource parameters in the big data code are as follows:

[0088] num-executors: representing the number of Executor processes for executing Spark tasks;

[0089] executor-cores: representing the number of processor cores occupied by each Executor process;

[0090] The number of processor cores determines the ability of each Executor to execute task threads in parallel, and the total number of processor cores occupied by the total Executor processes corresponding to the big data code can be determined according to the number of Executor processes and the number of processor cores occupied by each Executor process.

[0091] It should be noted that the number of processor cores can be determined by the maximum number of processor cores in the department resource queue.

[0092] It can be understood that if the department resource queue is shared with others, the number of processor cores can be determined as a certain percentage of the maximum number of processor cores in the department resource queue by setting the occupation ratio.

[0093] In some embodiments, the detection rule information further includes a data skew determination rule, see Figure 2 , Figure 2 is a flowchart of an automatic detection method of a big data code provided by another embodiment of the present application, as Figure 2 shown, after obtaining the preset detection rule information, the automatic detection method of the big data code further includes:

[0094] Step S210, detecting whether the big data code has data skew based on the data skew determination rule, and generating a data skew detection result corresponding to the big data code;

[0095] The resource detection result corresponding to the big data code is sent to the user, including:

[0096] The resource detection result corresponding to the big data code and the data skew detection result are sent to the user.

[0097] It should be understood that the preset detection rule information further includes data skew determination rule, and the data skew determination rule is used to indicate a data skew determination operation on the big data code. The data skew refers to that in a parallel computing mode (for example, in a map-reduce framework of Hadoop, data is divided into N segments and distributed to different computing nodes for separate calculation), a single computing node obtains and processes a data stream far greater than other nodes, causing excessive computing pressure of the node, resulting in a decline in computing efficiency or a computing memory overflow.

[0098] In some embodiments, referring to Figure 3 , Figure 3 is Figure 2 a sub-step flowchart of step S210 in Figure 3 As shown in the figure, the detection of whether the big data code has data skew based on the data skew determination rule to generate the data skew detection result corresponding to the big data code includes but is not limited to steps S310 to S330:

[0099] In step S310, the task execution information corresponding to the big data code is obtained through a resource management platform, and the task execution information includes the execution state and the processing data volume of each node task.

[0100] In step S320, the average processing data volume is determined according to the processing data volume of each node task.

[0101] In step S330, the data skew detection result corresponding to the big data code is generated based on the execution state of each node task and the size relationship between the processing data volume of each node task and the average processing data volume.

[0102] For example, in the execution process of the big data code, the reduce task execution information corresponding to the big data code is obtained through the Hadoop resource manager of Apache Hadoop YARN (Yet Another Resource Negotiator), including the execution state and the processing data volume of each reduce subtask.

[0103] Further, the average processing data amount of each node task is determined according to the processing data amount of each node task, and the data skew detection result corresponding to the big data code is generated based on the execution state of each node task and the size relationship between the processing data amount and the average processing data amount of each node task. Specifically, if it is found that a small number of reduce sub-tasks are not completed, the processing data amount corresponding to the incomplete reduce sub-task is then obtained and compared with the average processing data amount. If the processing data amount corresponding to the incomplete reduce sub-task is much larger than the average processing data amount, it can be determined that the writing of the big data code causes data skew.

[0104] In one specific embodiment, the data skew detection result includes the node task corresponding to the big data code, the processing data amount of each node task, and whether it causes data skew. The data skew detection result containing the above information is sent to the user to inform the user whether the writing of the big data code will cause data skew, thereby realizing the data skew determination of the big data code.

[0105] In some embodiments, referring to Figure 6 , Figure 6 is a flowchart of an automatic detection method of a big data code provided by an embodiment of the present application, as shown in Figure 6 After receiving the big data code uploaded by the user, the method further includes:

[0106] Step S610, keyword detection is performed on the big data code to determine whether a preset target keyword exists in the big data code.

[0107] Step S620, if it is determined that the preset target keyword exists in the big data code, the target keyword and the code statement containing the target keyword in the big data code are sent to the user.

[0108] For example, in order to prevent the occurrence of false deletion, a target keyword related to deletion operation such as "drop", "truncate" or "delete" can be preset, and keyword detection is performed on the big data code to determine whether the target keyword exists in the big data code. If it is determined that the target keyword exists in the big data code, the target keyword and the code statement containing the target keyword in the big data code are sent to the user, thereby realizing false deletion data warning. That is, the user is reminded that the code statement for deleting data exists in the big data code, and the user further checks whether the code statement has a problem. Specifically, the big data code can be stored in a preset database, and keyword query is performed through a query statement of the database.

[0109] It can be understood that before keyword detection is performed on the big data code, the target keyword configured by the user can be acquired, so as to realize the configurability of keyword detection.

[0110] In some embodiments, referring to Figure 7 , Figure 7 is a flowchart of a big data code automatic detection method provided by the embodiment of the application, as shown in Figure 7 After the big data code uploaded by the user is received, the method further includes:

[0111] Step S710, performing syntax detection on the big data code to determine whether the big data code has syntax errors;

[0112] Step S720, if it is determined that the big data code has syntax errors, sending the code statements with syntax errors in the big data code to the user.

[0113] In one specific embodiment, the receiving of the big data code uploaded by the user includes:

[0114] receiving the big data code uploaded by the user and a target syntax category;

[0115] The syntax detection on the big data code to determine whether the big data code has syntax errors includes:

[0116] acquiring a corresponding syntax detection rule according to the target syntax category, and performing syntax detection on the big data code based on the syntax detection rule to determine whether the big data code has syntax errors.

[0117] It can be understood that the target syntax category can be Spark or Scala. Spark and Scala have their own syntax rules, and the user specifies the language category (Spark or Scala) of the big data code when uploading the big data code, and then determines the corresponding syntax detection rule based on the target syntax category.

[0118] Specifically, a Python code programmed based on the syntax rules of Spark or Scala scripts is acquired, the big data code is searched through the Python code in multiple concurrent threads, if it is determined that the big data code has syntax errors, the code statements with syntax errors in the big data code are sent to the user, and syntax error alarm is realized.

[0119] It should be noted that when detecting the big data code, the specific information of this detection can be recorded, for example, the interface time, content, input parameters and business trip information are recorded one by one. On the one hand, it can support the user to view the detection information related to the big data code, and on the other hand, in this detection process, the detection information of the same segment of the big data code of the same user in the last time can be obtained. By comparing the detection information of two times, it can be determined whether the code defects of the big data code in the last detection are modified and solved, and the user can also be warned of the uncertainty risk, for example, the big data code is detected as 4 code structures in the last scan, and 5 code structures in this scan. Therefore, it can be warned whether a segment of code is mistakenly deleted.

[0120] The automatic detection method of the big data code provided by the embodiment of the application is described below through a specific embodiment:

[0121] The method is applied to a code self-service scanning platform. The user uploads the big data code to be handed over and the language type of the big data code to the scanning platform. The scanning platform first scans and detects the target keyword related to the deletion operation in the big data code to prevent the occurrence of the case of mistakenly deleting data. Then, the corresponding syntax rule is determined according to the language type of the big data code, and the big data code is scanned and detected based on the syntax rule. The user is notified of the code statement with syntax errors in the big data code, preventing errors in the test environment. Next, the scanning platform obtains the preconfigured detection rule information, and detects the big data code based on the detection rule information, realizes the detection operation of the memory resource limitation, the processor core resource limitation alarm and the data skew determination in the big data code. At the same time, the detection results of this scan are recorded, and the detection results of two scans are compared in the next scan, which can prompt the user of the uncertainty risk of the big data code.

[0122] Through the scanning and detection of the code self-service scanning platform on the big data code to be handed over, the defects existing in the big data code can be found in advance before the formal code handover stage, thereby improving the quality of the handed-over code and the work efficiency of the test.

[0123] The application provides an automatic detection method for big data code. The automatic detection method for big data code receives big data code uploaded by a user, acquires preset detection rule information, and detects the big data code according to the detection rule information. The detection rule information is used to indicate the detection operation on the big data code, and includes resource limitation occupied by total processes corresponding to the big data code. Then, preset job process resource parameters in the big data code are acquired, and the preset job process resource parameters in the big data code are detected to determine whether the resource limitation occupied by the total processes corresponding to the big data code is exceeded, and a resource detection result corresponding to the big data code is generated. Finally, the resource detection result corresponding to the big data code is sent to the user. According to the application, the preset detection rule information is acquired, automatic detection of the big data code uploaded by the user is realized, defects in the big data code can be detected, and human and resource costs are reduced.

[0124] See Figure 8 The application also provides an automatic detection device 100 for big data code, which comprises:

[0125] A receiving module 110 is configured to receive big data code uploaded by a user.

[0126] A first acquiring module 120 is configured to acquire preset detection rule information. The detection rule information includes resource limitation occupied by total processes corresponding to the big data code.

[0127] A second acquiring module 130 is configured to acquire preset job process resource parameters in the big data code.

[0128] A detection module 140 is configured to detect the preset job process resource parameters in the big data code to determine whether the resource limitation occupied by the total processes corresponding to the big data code is exceeded, and generate a resource detection result corresponding to the big data code.

[0129] A result sending module 150 is configured to send the resource detection result corresponding to the big data code to the user.

[0130] In some embodiments, the detection rule information further includes data skew determination rules. The detection module 140 is further configured to detect whether the big data code has data skew based on the data skew determination rules, and generate a data skew detection result corresponding to the big data code.

[0131] In some embodiments, the detection module 140 further comprises:

[0132] The first detection submodule 141 is configured to acquire task execution information corresponding to the big data code through the resource management platform, wherein the task execution information comprises an execution state of each node task and a processing data volume of each node task.

[0133] The second detection submodule 142 is configured to determine an average processing data volume according to the processing data volume of each node task.

[0134] The third detection submodule 143 is configured to generate a data skew detection result corresponding to the big data code based on the execution state of each node task and a size relationship between the processing data volume of each node task and the average processing data volume.

[0135] In some embodiments, the automatic detection device 100 for the big data code further comprises:

[0136] The keyword detection module 160 is configured to perform keyword detection on the big data code to determine whether a preset target keyword exists in the big data code.

[0137] The first sending module 170 is configured to send the target keyword and a code statement containing the target keyword in the big data code to the user if it is determined that the preset target keyword exists in the big data code.

[0138] In some embodiments, the automatic detection device 100 for the big data code further comprises:

[0139] The syntax detection module 180 is configured to perform syntax detection on the big data code to determine whether a syntax error exists in the big data code.

[0140] The second sending module 190 is configured to send a code statement in which a syntax error exists in the big data code to the user if it is determined that the syntax error exists in the big data code.

[0141] The application provides an automatic detection device for big data code. The automatic detection device receives a big data code uploaded by a user through a receiving module, and acquires preset detection rule information through a first acquiring module. The detection rule information is used to indicate a detection operation performed on the big data code. The detection rule information includes a resource limit occupied by total processes corresponding to the big data code. Then, a second module is used to acquire preset job process resource parameters in the big data code. A detection module is used to detect the preset job process resource parameters in the big data code, so as to determine whether the resource occupied by the total processes corresponding to the big data code exceeds the resource limit, and generate a resource detection result corresponding to the big data code. Finally, a result sending module is used to send the resource detection result corresponding to the big data code to the user. According to the application, the preset detection rule information is acquired, the automatic detection of the big data code uploaded by the user is realized, the defects in the big data code can be detected, and the labor and resource costs can be reduced.

[0142] It should be noted that the information interaction and execution process between the modules of the device are based on the same concept as the method embodiments of the application, and the specific functions and technical effects brought by the method embodiments can be referred to the method embodiments part. Here, no longer be repeated.

[0143] Please refer to Figure 9 , Figure 9 A hardware structure of an electronic device is shown, and the electronic device comprises:

[0144] The processor 210 can be implemented in the form of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related computer programs to implement the technical solutions provided by the application.

[0145] The memory 220 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 220 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 220 and are called and executed by the processor 210 to implement the automatic detection method of the big data code.

[0146] The input / output interface 230 is used to realize information input and output.

[0147] The communication interface 240 is configured to realize the communication interaction between the device and other devices, and the communication can be realized through a wired manner (for example, a USB, a network cable, and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, and the like). The bus 250 is configured to transmit information between each component (for example, the processor 210, the memory 220, the input / output interface 230, and the communication interface 240) of the device.

[0148] The processor 210, the memory 220, the input / output interface 230, and the communication interface 240 are connected to each other through the bus 250.

[0149] The embodiment of the present application further provides a storage medium, the storage medium is a computer readable storage medium, and the storage medium is used for computer readable storage. The storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to realize the automatic detection method of the big data code.

[0150] The memory is a computer readable storage medium, and can be used to store software programs and computer executable programs. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0151] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0152] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to realize the purpose of the embodiments.

[0153] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0154] In the above embodiments, the description of each embodiment is focused on, and the part not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0155] The terms "first", "second", "third", "fourth" etc. (if any) in the description of the application and in the above drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0156] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0157] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or can be combined into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0158] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0159] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0160] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical scheme of the present application or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0161] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for automatic detection of big data code, characterized in that, The method comprises: receiving a large data code uploaded by a user; obtaining preset detection rule information, wherein the detection rule information comprises a resource limit occupied by total processes corresponding to the large data code; obtaining a preset job process resource parameter in the large data code; detecting the preset job process resource parameter in the large data code to determine whether the resource occupied by the total processes corresponding to the large data code exceeds the resource limit, and generating a resource detection result corresponding to the large data code; sending the resource detection result corresponding to the large data code to the user; the resource limit comprises a memory capacity limit, and the job process resource parameter comprises a job process quantity and a content capacity occupied by each job process; the detection of the preset job process resource parameter in the large data code to determine whether the resource occupied by the total processes corresponding to the large data code exceeds the resource limit, and the generation of the resource detection result corresponding to the large data code, comprise: determining the memory capacity occupied by the total processes corresponding to the large data code according to the job process quantity and the memory capacity occupied by each job process; generating the resource detection result corresponding to the large data code based on the size relationship between the memory capacity occupied by the total processes corresponding to the large data code and the memory capacity limit.

2. The method of claim 1, wherein, The detection rule information further comprises a data skew determination rule; after obtaining the preset detection rule information, the method further comprises: detecting whether the large data code has data skew based on the data skew determination rule, and generating a data skew detection result corresponding to the large data code; the sending of the resource detection result corresponding to the large data code to the user comprises: sending the resource detection result corresponding to the large data code and the data skew detection result to the user.

3. The method of claim 2, wherein, The detection of whether the large data code has data skew based on the data skew determination rule, and the generation of the data skew detection result corresponding to the large data code, comprise: obtaining task execution information corresponding to the large data code through a resource management platform, wherein the task execution information comprises an execution state of each node task and a processing data quantity; determining an average processing data quantity according to the processing data quantity of each node task; generating the data skew detection result corresponding to the large data code based on the size relationship between the execution state of each node task and the processing data quantity of each node task and the average processing data quantity.

4. The method of claim 1, wherein, After receiving the large data code uploaded by the user, the method further comprises: performing syntax detection on the large data code to determine whether the large data code has syntax errors; if it is determined that the large data code has syntax errors, sending code statements with syntax errors in the large data code to the user.

5. The method of claim 1, wherein, After receiving the large data code uploaded by the user, the method further comprises: performing keyword detection on the large data code to determine whether a preset target keyword exists in the large data code; If it is determined that the target keyword exists in the big data code, the target keyword and a code statement containing the target keyword in the big data code are sent to the user.

6. A method for automatic detection of big data code, characterized in that, The method comprises: receiving a big data code uploaded by a user; obtaining preset detection rule information, the detection rule information comprising a resource limit occupied by total processes corresponding to the big data code; obtaining a preset job process resource parameter in the big data code; detecting the preset job process resource parameter in the big data code to determine whether the resource occupied by the total processes corresponding to the big data code exceeds the resource limit, and generating a resource detection result corresponding to the big data code; sending the resource detection result corresponding to the big data code to the user; wherein the resource limit comprises a processor core quantity limit, the job process resource parameter comprises a job process quantity and a processor core quantity occupied by each job process; the detection of the preset job process resource parameter in the big data code to determine whether the resource occupied by the total processes corresponding to the big data code exceeds the resource limit, and the generation of the resource detection result corresponding to the big data code, comprise: determining a total processor core quantity occupied by the total processes corresponding to the big data code according to the job process quantity and the processor core quantity occupied by each job process; generating the resource detection result of the big data code based on the size relationship between the total processor core quantity occupied by the total processes corresponding to the big data code and the processor core quantity limit.

7. An apparatus for automatic detection of big data code, characterized by, The apparatus comprises: a receiving module configured to receive a big data code uploaded by a user; a first obtaining module configured to obtain preset detection rule information, the detection rule information comprising a resource limit occupied by total processes corresponding to the big data code; a second obtaining module configured to obtain a preset job process resource parameter in the big data code; a detection module configured to detect the preset job process resource parameter in the big data code to determine whether the resource occupied by the total processes corresponding to the big data code exceeds the resource limit, and generate a resource detection result corresponding to the big data code; a result sending module configured to send the resource detection result corresponding to the big data code to the user; wherein the resource limit comprises a processor core quantity limit, the job process resource parameter comprises a job process quantity and a processor core quantity occupied by each job process; the detection of the preset job process resource parameter in the big data code to determine whether the resource occupied by the total processes corresponding to the big data code exceeds the resource limit, and the generation of the resource detection result corresponding to the big data code, comprise: determining a total processor core quantity occupied by the total processes corresponding to the big data code according to the job process quantity and the processor core quantity occupied by each job process; generating the resource detection result of the big data code based on the size relationship between the total processor core quantity occupied by the total processes corresponding to the big data code and the processor core quantity limit. The resource detection result of the big data code is generated based on a size relationship between a number of processor cores occupied by total processes corresponding to the big data code and a size of the number of processor cores.

8. An apparatus for automatic detection of big data code, characterized by, The apparatus comprises: a receiving module configured to receive a big data code uploaded by a user; a first obtaining module configured to obtain preset detection rule information, the detection rule information comprising a resource limit occupied by total processes corresponding to the big data code; a second obtaining module configured to obtain preset job process resource parameters in the big data code; a detection module configured to detect the preset job process resource parameters in the big data code to determine whether a resource occupied by the total processes corresponding to the big data code exceeds the resource limit, and to generate a resource detection result corresponding to the big data code; a result sending module configured to send the resource detection result corresponding to the big data code to the user; The resource limit comprises a memory capacity limit, and the job process resource parameters comprise a number of job processes and a content capacity occupied by each job process; the detection module is configured to detect the preset job process resource parameters in the big data code to determine whether a resource occupied by the total processes corresponding to the big data code exceeds the resource limit, and to generate a resource detection result corresponding to the big data code, comprising: determining a memory capacity occupied by the total processes corresponding to the big data code according to the number of job processes and the memory capacity occupied by each job process; generating the resource detection result corresponding to the big data code based on a size relationship between the memory capacity occupied by the total processes corresponding to the big data code and the memory capacity limit.

9. An electronic device, comprising: comprise: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program, and the computer program is executed by the at least one processor to enable the at least one processor to execute the automatic detection method of the big data code according to any one of claims 1 to 6.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the automatic detection method of the big data code according to any one of claims 1 to 6.

Images