Code debugging method and device, storage medium and electronic equipment

Abstract

The application discloses a code debugging method and device, a storage medium and an electronic device. The method comprises the following steps: obtaining a first code set to be run; scanning the first code set to obtain a target scanning result; in the case that the target scanning result indicates that there is a first code subset to be optimized in the first code set, replacing the first code subset in the first code set with a target code subset to obtain a second code set, wherein the second code set comprises the target code subset and codes in the first code set except the first code subset, the first code subset and the target code subset have the same function, and the running resources required for running the target code subset are less than the running resources required for running the first code subset; and running the second code set in a target computing platform. The application solves the technical problem of low code optimization efficiency in the related art.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a code debugging method and device, a storage medium, and an electronic device. Background Art

[0002] Current technologies generally require users to understand the operating mechanisms of certain computing platforms and the meaning of various parameters in order to achieve performance tuning during code debugging. Furthermore, some performance tuning methods can only address specific performance issues, which is limited. These methods automatically adjust fixed parameters such as memory size and number of CPU cores, but cannot provide optimization suggestions for specific code issues. For example, during the Spark computing platform's calculation process, since performance tuning for the Spark computing platform is multifaceted, it is necessary to manually understand the specific meaning of all code and manually optimize the code. For example, in the written code, the result of each operation is assigned to a new RDD, resulting in duplicate RDD creation. This requires a user to understand the special effects of the Spark computing platform, understand and discover the assignment code, and then manually optimize the code. This significantly reduces the efficiency of code optimization and increases the time required.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present invention provide a code debugging method and apparatus, a storage medium, and an electronic device, so as to at least solve the technical problem of low efficiency of code optimization existing in the related art.

[0005] According to one aspect of an embodiment of the present invention, a code debugging method is provided, comprising: obtaining a first code set to be run; scanning the first code set to obtain a target scanning result, wherein the target scanning result is used to indicate whether a first code subset to be optimized exists in the first code set; when the target scanning result indicates that the first code subset to be optimized exists in the first code set, replacing the first code subset in the first code set with a target code subset to obtain a second code set, wherein the second code set includes the target code subset and code in the first code set other than the first code subset, the first code subset and the target code subset have the same function, and running resources required to run the target code subset are less than running resources required to run the first code subset; and running the second code set in a target computing platform.

[0006] According to another aspect of an embodiment of the present invention, a code debugging device is further provided, including: an acquisition module for acquiring a first code set to be run; a scanning module for scanning the first code set to obtain a target scanning result, wherein the target scanning result is used to indicate whether a first code subset to be optimized exists in the first code set; an optimization module for replacing the first code subset in the first code set with a target code subset to obtain a second code set when the target scanning result indicates that the first code subset to be optimized exists in the first code set, wherein the second code set includes the target code subset and code in the first code set other than the first code subset, the first code subset and the target code subset have the same function, and the running resources required to run the target code subset are less than the running resources required to run the first code subset; and a running module for running the second code set in a target computing platform.

[0007] Optionally, the scanning module includes: a first scanning module, used to generate the target scanning result when a first code subset and a second code subset exist in the first code set, wherein the target scanning result is used to indicate that the first code subset to be optimized exists in the first code set, the first code subset is used to load target data into the created first data set, the second code subset is used to load the target data into the created second data set, and in the first code set, the second code subset is executed earlier than the first code subset.

[0008] Optionally, the optimization module includes: a first optimization unit, used to delete the first code in the first code subset and modify the second code into a first target code when the first code subset includes a first code and a second code, wherein the first code is used to load the target data into the first data set, the second code is used to perform a target operation on the target data in the first data set, and the first target code is used to perform the target operation on the target data in the second data set.

[0009] Optionally, the scanning module includes: a second scanning module, used to generate the target scanning result when there are a first code subset and a second code subset in the first code set, wherein the target scanning result is used to indicate that there is the first code subset to be optimized in the first code set, the first code subset is used to execute a first group of operation operations in sequence, the second code subset is used to execute a second group of operation operations in sequence, a first part of the operation operations in the first group of operation operations and a second part of the operation operations in the second group of operation operations are the same, and the second code subset is executed earlier than the first code subset in the first code set; wherein the first part of the operation operations includes a plurality of consecutive operation operations starting from the first operation operation in the first group of operation operations, and the second part of the operation operations includes a plurality of consecutive operation operations starting from the first operation operation in the second group of operation operations; or, the first part of the operation operations includes the first operation operation in the first group of operation operations, and the second part of the operation operations includes the first operation operation in the second group of operation operations.

[0010] Optionally, the optimization module includes: a second optimization unit, used to delete the code corresponding to the first part of the operation in the first code subset, and modify the code corresponding to the next operation of the first part of the operation to a second target code, wherein the second target code is used to obtain the operation results of the second group of operation operations, and execute the next operation on the obtained operation results.

[0011] Optionally, the optimization module includes: a third optimization unit, used to add a third target code to the second code subset, and modify the code corresponding to the next operation of the second part of the operation operation to a fourth target code, wherein the third target code is set to be executed after the code corresponding to the second part of the operation operation and before the fourth target code, the third target code is used to store the operation results of the second group of operation operations in the target cache, the fourth target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the second part of the operation operation on the obtained operation results of the second group of operation operations, and the second target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the first part of the operation operation on the obtained operation results.

[0012] Optionally, the scanning module includes: a third scanning module, used to generate the target scanning result when a first code subset exists in the first code set, wherein the target scanning result is used to indicate that the first code subset to be optimized exists in the first code set, and the first code subset is used to transmit data in a first data group to a target data processing node, and the first data group is allowed to include multiple identical data.

[0013] Optionally, the optimization module includes: a fourth optimization unit, used to replace the first code subset with the target code subset in the first code set, and the target code subset is used to aggregate the multiple identical data in the first data group into one data when the first data group includes multiple identical data, to obtain a second data group, and transmit the second data group to the target node.

[0014] Optionally, the device further includes at least one of the following: a first prompt module, configured to generate first prompt information when the target scanning result indicates that a first code subset to be optimized exists in the first code set, wherein the first prompt information is used to indicate a position of the first code subset in the first code set; and a second prompt module, configured to generate second prompt information when the target scanning result indicates that a first code subset to be optimized exists in the first code set, wherein the second prompt information is used to indicate whether the first code subset needs to be replaced with the target code subset.

[0015] Optionally, the optimization module includes: a generation unit for generating a target optimization plan when the target scanning result indicates that there is a first code subset to be optimized in the first code set, wherein the target optimization plan is used to replace the first code subset with the second code subset; an acquisition unit for acquiring a target execution instruction, wherein the target execution instruction is used to instruct the execution of the target optimization plan; and an execution unit for executing the target optimization plan in response to the target execution instruction to replace the first code subset in the first code set with the second code subset to obtain the second code set.

[0016] Optionally, the device is also used to: after running the second code set in the target computing platform, when the second code set completes running on the target computing platform, generate an operation log of the second code set, wherein the operation log records at least one of the following: the operating parameters of the target computing platform during the process of running the second code set in the target computing platform, and the interaction parameters of the target data operated by the second code set during the process of running the second code set in the target computing platform.

[0017] Optionally, after generating the operation log of the second code set, the device further includes at least one of the following: a first processing module, configured to add a first segment of code to the second code set when the loading duration represented by the operation parameter is greater than or equal to a preset duration threshold, wherein the loading duration is the duration for loading the target data when the target computing platform runs the second code set, and the first segment of code is used to increase the capacity of each partition in the partition set of the target computing platform, or to increase the capacity of each partition in the partition set of the target computing platform and reduce the number of partitions in the partition set, and the target computing platform is used to load the target data from the partition set; a second A processing module for adding a second segment of code to the second code set when the value of the memory occupancy parameter represented by the operating parameter is greater than or equal to a preset occupancy threshold, wherein the value of the memory occupancy parameter is used to represent the memory occupied by the target computing platform when running the second code set to process the target data, and the second segment of code is used to increase the memory allowed to be used by the target computing platform; a third processing module for adding a third segment of code to the second code set when the second code set includes code for performing transmission operations or collection operations on part or all of the target data, wherein the third segment of code is used to increase the memory allowed to be used by the target computing platform.

[0018] Optionally, the device is further used to: when the operating parameter indicates that data skew occurs during the operation of the second code set, determine whether the second code set includes code for performing a join operation on a table formed by part or all of the data in the target data; when the second code set includes the code for performing a join operation on a table formed by part or all of the data in the target data, search for a first data set and a second data set in the target data, wherein the magnitude of the data in the first data set is smaller than a preset first data magnitude, and the data in the first data set is set to perform the join operation, the magnitude of the data in the second data set is greater than a preset second data magnitude, and the data in the second data set is set to perform the join operation, and the first data magnitude is smaller than the second data magnitude; after finding the first data set, In the case of one data set, the first data set is broadcast to multiple data processing nodes, so that the multiple data processing nodes perform the join operation on the table formed by the data in the first data set in parallel to obtain a first operation result; when the second data set is found, the data in the second data set is split into a first data subset with a data skew identifier and a second data subset without a data skew identifier; the first data subset is broadcast to the multiple data processing nodes, so that the multiple data processing nodes perform the join operation on the table formed by the data in the first data subset in parallel to obtain a second operation result; the join operation is directly performed on the table formed by the data in the second data subset to obtain a third operation result; the second operation result and the third operation result are merged to obtain a fourth operation result.

[0019] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned code debugging method when running.

[0020] According to another aspect of an embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the code debugging method through the computer program.

[0021] In an embodiment of the present invention, without changing the functionality of a code collection, the code collection is scanned before it is run on a computing platform to identify the code to be optimized within the code collection. This code is then optimized. This solves the problem of low code optimization efficiency caused by manual code optimization. Furthermore, because the optimized code requires fewer runtime resources than the unoptimized code, this saves resources on the computing platform, further resolving the technical issue of low code optimization efficiency that exists in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0023] Figure 1 is a schematic diagram of an application environment of an optional code debugging method according to an embodiment of the present invention;

[0024] Figure 2 is a flowchart of an optional code debugging method according to an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of an optional code debugging method according to an embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0027] Figure 5 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0028] Figure 6 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0029] Figure 7 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0030] Figure 8 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0031] Figure 9 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0032] Figure 10 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0033] Figure 11is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0034] Figure 12 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0035] Figure 13 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0036] Figure 14 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention;

[0037] Figure 15 This is a schematic structural diagram of an optional code debugging method and device according to an embodiment of the present invention;

[0038] Figure 16 FIG. 4 is a schematic structural diagram of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0042] RDD: Resilient Distributed Dataset, elastic distributed dataset.

[0043] Spark platform, a fast and general computing engine designed for large-scale data processing.

[0044] Shuffle: A function that rearranges the elements of an array into a random order.

[0045] The present invention will be described below in conjunction with embodiments:

[0046] According to one aspect of an embodiment of the present invention, a code debugging method is provided. Optionally, in this embodiment, the code debugging method can be applied to Figure 1 In the hardware environment composed of server 101 and terminal 103 shown in FIG. Figure 1 As shown, the server 101 is connected to the terminal 103 via a network and can be used to provide services to the terminal or a client installed on the terminal. The client can be a video client, an instant messaging client, a browser client, an education client, a game client, etc. A database 105 can be set on the server or independently of the server to provide data storage services for the server 101, for example, a server for storing big data. The above-mentioned network can include but is not limited to: a wired network and a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that implement wireless communication. The terminal 103 can be a terminal configured with an application connected to a computing platform, and can include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Device), a PAD, a desktop computer, a smart TV and other computer devices. The above-mentioned server can be a single server, a server cluster consisting of multiple servers, or a cloud server, and can include but is not limited to a router or a gateway. The application using the above-mentioned code debugging method is displayed through the client 107. The above is only an example and is not limited to this embodiment.

[0047] Alternatively, as an optional implementation, Figure 2 As shown, the above code debugging method includes:

[0048] S202, obtaining a first code set to be executed;

[0049] S204, scanning the first code set to obtain a target scanning result, wherein the target scanning result is used to indicate whether a first code subset to be optimized exists in the first code set;

[0050] S206: If the target scanning result indicates that the first code subset to be optimized exists in the first code set, replace the first code subset in the first code set with the target code subset to obtain a second code set, wherein the second code set includes the target code subset and codes in the first code set other than the first code subset, the first code subset and the target code subset have the same function, and running resources required to run the target code subset are less than running resources required to run the first code subset;

[0051] S208: Run the second code set in the target computing platform.

[0052] Optionally, in this embodiment, Figure 3 is a schematic diagram of a code debugging method according to an embodiment of the present invention. The code debugging method may include but is not limited to being applied to Figure 3 In the architecture shown, the target computing platform 302 is used to pull the data to be processed from the data processing node 304 and perform debugging based on the debugging code sent by the client 306. Figure 3 As shown, the above code debugging method can be implemented on the client 306 of the server side through the following steps:

[0053] Obtaining a first code set to be executed in the client 306 on the server side;

[0054] Scanning the first code set in the client 306 on the server side to obtain a target scanning result, wherein the target scanning result is used to indicate whether the first code subset to be optimized exists in the first code set;

[0055] In the client 306 on the server side, if the target scanning result indicates that the first code subset to be optimized exists in the first code set, the first code subset in the first code set is replaced with the target code subset to obtain a second code set, wherein the second code set includes the target code subset and codes in the first code set other than the first code subset, the first code subset and the target code subset have the same function, and running resources required to run the target code subset are less than running resources required to run the first code subset;

[0056] The second code set is submitted to the target computing platform 302 in the client 306 on the server side, so that the second code set is run in the target computing platform 302 .

[0057] The above is only an example and is not specifically limited in this embodiment.

[0058] Optionally, in this embodiment, it may include but is not limited to implementation based on cloud technology.

[0059] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data calculation, storage, processing and sharing.

[0060] Cloud technology is a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Different levels of data will be processed separately, and data from various industries will require a strong system backend, which can only be achieved through cloud computing.

[0061] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, these resources appear infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a per-use basis.

[0062] As a provider of cloud computing infrastructure, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0063] Based on logical functional divisions, the PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. SaaS can also be deployed directly on top of IaaS. PaaS is a platform for software execution, such as databases and web containers. SaaS is a variety of business software, such as web portals and text messaging tools. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0064] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0065] Currently, storage systems utilize a method for creating logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a specific storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID) of the data entity. The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can provide access to the data based on the storage location information of each object.

[0066] The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0067] Big data refers to collections of data that cannot be captured, managed, and processed within a specific timeframe using conventional software tools. These massive, rapidly growing, and diverse information assets require new processing models to enhance decision-making, insight discovery, and process optimization. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to efficiently process large amounts of time-sensitive data. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the internet, and scalable storage systems.

[0068] Optionally, in this embodiment, the application scenarios of the above code debugging method may include but are not limited to medical care, finance, credit reporting, banking, government affairs, government, games, energy, education, security, buildings, games, transportation, Internet of Things, industry and other application scenarios that require the use of a target computing platform for data processing, for example, the big data processing scenarios corresponding to the above business scenarios.

[0069] Taking the financial business scenario as an example, the above-mentioned first code set may include but is not limited to a code set for performing big data processing in a target computing platform. The above-mentioned target computing platform may include but is not limited to a Spark computing platform for big data processing, etc., and may include but is not limited to a cloud computing platform.

[0070] For example, Figure 4 FIG. 1 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention. Figure 4 As shown, the above code debugging method is applied to a financial services platform 402 as an example, wherein the financial services platform 402 includes an application program that uses a Spark platform 404 for data processing. First, a code set to be run in the Spark platform 404 (corresponding to the aforementioned first code set) is obtained, wherein the code set contains a first code subset to be optimized. Second, the code set is scanned, and a target scanning result is displayed on a display interface 408 of a client 406, wherein the target scanning result displays the position of the first code subset to be optimized in the code set and an automatically generated optimization suggestion for the code subset. Third, an interactive operation is obtained to optimize the first code subset and replace it with the target code subset to obtain a second code set. Finally, the optimized second code set is submitted to the Spark platform for execution to realize financial data processing. During the financial data processing, financial data such as account information and transaction information can be processed by the second code set in the Spark computing platform and stored in an RDD data set 410.

[0071] For example, Figure 5 FIG. 1 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention. Figure 5As shown, the above code debugging method is applied to a game business platform 502 as an example, wherein the game business platform 502 includes an application program that uses a Spark platform 504 for data processing. First, a code set to be run in the Spark platform 504 (corresponding to the aforementioned first code set) is obtained, wherein the code set contains a code subset to be optimized. Second, the code set is scanned, and a target scanning result is displayed on a display interface 508 of a client 506. The target scanning result displays the location of the first code subset to be optimized in the code set and an automatically generated optimization suggestion for the first code subset to be optimized. Third, an interactive operation is obtained to optimize the first code subset and replace it with the target code subset to obtain a second code set. Finally, the optimized second code set is submitted to the Spark platform for execution to process game data. During the game data processing, game data such as account information, game operation data, and game results can be processed by the second code set in the Spark computing platform and stored in an RDD data set 510.

[0072] The above is only an example and this embodiment does not impose any specific limitation.

[0073] Optionally, in this embodiment, the above-mentioned scanning of the first code set may include but is not limited to static scanning, dynamic scanning, etc. Taking static scanning as an example, the above-mentioned first code set may be statically scanned to obtain a corresponding static target scanning result. The above-mentioned target scanning result may include but is not limited to an indication of whether there is a first code subset to be optimized in the above-mentioned first code set.

[0074] Specifically, it may include but is not limited to indicating whether there are repeatedly created data sets in the above-mentioned first code set, and may also include but is not limited to indicating whether the code in the above-mentioned first code set is used to perform repeated operator operations, and may also include but is not limited to whether the operator used by the code in the above-mentioned first code set is an operator with better performance.

[0075] Optionally, in this embodiment, the above-mentioned first code subset represents the first code subset to be optimized in the above-mentioned first code set. By replacing the above-mentioned first code subset with the target code subset, the first code set can be optimized into the second code set, and then, the second code set can be submitted to the target computing platform to run the above-mentioned second code set.

[0076] Taking the example of whether there is a duplicated dataset in the first code set, the code debugging method may include but is not limited to the following:

[0077] The first set of code above, before scanning, requires creating multiple RDDs:

[0078] val rdd1=sc.textFile("hello.txt")

[0079] rdd1.map(...)

[0080] val rdd2=sc.textFile("hello.txt")

[0081] rdd2.reduce(...)

[0082] The second set of codes after scanning can reuse RDD:

[0083] val rdd1=sc.textFile("hello.txt")

[0084] rdd1.map(...)

[0085] rdd1.reduce(...)

[0086] It should be noted that the above val rdd1=sc.textFile("hello.txt") means creating rdd1, valrdd2=sc.textFile("hello.txt") means creating rdd2, rdd1.map(...) and rdd2.reduce(...) are both a type of operation. When scanning the above first code set to load the same value into different RDDs, the codes assigned to rdd are merged, and the above different operations are performed in sequence. Therefore, since only one RDD needs to be created, repeated creation of RDD is avoided. Therefore, the running resources required to run the above target code subset are less than the running resources required to run the above first code subset.

[0087] Optionally, in this embodiment, the first code subset and the target code subset have the same function, which may include but is not limited to using operators with the same function to perform calculation operations, or the results of calculation operations performed by different code subsets are the same.

[0088] This embodiment obtains a first code set to be executed, scans the first code set, and obtains a target scan result. If the target scan result indicates that a first code subset to be optimized exists in the first code set, the first code subset in the first code set is replaced with the target code subset to obtain a second code set. The second code set includes the target code subset and code in the first code set other than the first code subset. The first code subset and the target code subset have the same functionality. The target code subset requires fewer runtime resources than the first code subset. The second code set is executed on a target computing platform. Without changing the functionality of the code set, the code set is scanned before the computing platform executes the code set to identify the code to be optimized in the code set and optimize the code to be optimized. This solves the problem of low code optimization efficiency caused by manual code optimization. Furthermore, because the optimized code requires fewer runtime resources than the unoptimized code, code runtime resources on the computing platform are saved, thereby solving the technical problem of low code optimization efficiency in related technologies.

[0089] As an optional solution, the first code set is scanned to obtain a target scanning result, including:

[0090] In a case where a first code subset and a second code subset exist in a first code set, a target scan result is generated, wherein the target scan result is used to indicate that a first code subset to be optimized exists in the first code set, the first code subset is used to load target data into a created first data set, the second code subset is used to load target data into a created second data set, and the second code subset is executed earlier than the first code subset in the first code set.

[0091] Optionally, in this embodiment, Figure 6 FIG. 1 is a schematic diagram of another code debugging method according to an embodiment of the present invention. Figure 6 The steps specifically include but are not limited to the following:

[0092] S1, the first code set and the interactive object "Scan" for scanning are displayed on the client;

[0093] S2, “scan” the interactive object to perform the interactive operation;

[0094] S3, in response to the above interaction operation, scanning the above first code set using a code scanning module;

[0095] S4, after the scan is completed, the target scan result is generated and displayed in the client.

[0096] The target scanning result is used to indicate that there is a first code subset to be optimized in the first code set. The target scanning result may include but is not limited to identifying the position of the first code subset to be optimized in the first code set and the reason for optimization, for example, Figure 6 The sign displayed in the "Duplicate! Optimize?"

[0097] For example, the first set of code above, before scanning, requires creating multiple RDDs:

[0098] val rdd1=sc.textFile("hello.txt") corresponds to the code in the second code subset;

[0099] rdd1.map(...) corresponds to the code in the second code subset;

[0100] val rdd2=sc.textFile("hello.txt") corresponds to the first code in the first code subset;

[0101] rdd2.reduce(...) corresponds to the second code in the first code subset;

[0102] The second set of codes, after scanning and optimization, can reuse RDD:

[0103] val rdd1=sc.textFile("hello.txt") corresponds to the code in the second code subset;

[0104] rdd1.map(...) corresponds to the code in the second code subset;

[0105] rdd1.reduce(...) corresponds to the first target code in the target code subset.

[0106] The above is only an example, and this embodiment does not impose any specific limitation.

[0107] Through this embodiment, a target scanning result is generated when a first code subset and a second code subset exist in a first code set, wherein the target scanning result includes a target scanning result, the target scanning result is used to indicate that a first code subset to be optimized exists in the first code set, the first code subset is used to load target data into a created first data set, and the second code subset is used to load target data into a created second data set. In the first code set, the second code subset is run earlier than the first code subset. Before the computing platform runs the code set, the code set is scanned to find out the code in the code set that repeatedly creates RDDs, and the code to be optimized is optimized, thereby avoiding the problem of easily repeatedly creating RDDs in manually optimized codes, thereby achieving the purpose of optimizing the code set, thereby achieving the technical effect of reducing the debugging time cost of code debugging on the computing platform and improving the debugging efficiency of code debugging on the computing platform, thereby solving the technical problem of low efficiency of code optimization in related technologies.

[0108] As an optional solution, the first code subset in the first code set is replaced with the target code subset to obtain a second code set, including:

[0109] In the case where the first code subset includes a first code and a second code, the first code is deleted in the first code subset and the second code is modified to a first target code, wherein the first code is used to load target data into the first data set, the second code is used to perform a target operation on the target data in the first data set, and the first target code is used to perform a target operation on the target data in the second data set.

[0110] Optionally, in this embodiment, the above-mentioned first code and the above-mentioned second code may include but are not limited to creating a target data set, and loading the above-mentioned target data into the target data set to perform the above-mentioned target operation. The above-mentioned first data set and the above-mentioned second data set may include but are not limited to RDD data sets used in the Spark platform, and the above-mentioned target data is the data loaded into the RDD.

[0111] For example, when the above-mentioned code debugging method is applied to a scenario of financial services, the above-mentioned first data set and the above-mentioned second data set are used to store financial data to be processed, including but not limited to account information, transaction history data, transaction amount, transaction time, etc. When the above-mentioned code debugging method is applied to a scenario of gaming business, the above-mentioned first data set and the above-mentioned second data set are used to store game data to be processed, including but not limited to game account information, game operation data, game transaction data, game image data, etc.

[0112] by Figure 6 For example, the above code debugging method may include but is not limited to the following:

[0113] S1, the first code set and the interactive object "Optimization" for scanning are displayed on the client;

[0114] S2, performs interactive operations on the interactive object “optimize”;

[0115] S3, in response to the above interactive operation, deleting the first code in the first code subset, and modifying the second code to the first target code.

[0116] The target data is "hello.txt". By running the first code, the hello.txt can be loaded into rdd1 (corresponding to the first data set mentioned above).

[0117] The above is only an example, and this embodiment does not impose any specific limitation.

[0118] Through this embodiment, when the first code subset includes the first code and the second code, the first code is deleted in the first code subset and the second code is modified to the first target code, wherein the first code is used to load the target data into the first data set, the second code is used to perform the target operation on the target data in the first data set, and the first target code is used to perform the target operation on the target data in the second data set. By deleting the first code and modifying the second code to the first target code, the code logic of repeatedly creating RDD is optimized, which facilitates the optimization of the code logic of repeatedly creating RDD, avoids the technical problem of repeatedly creating RDD, and thus achieves the purpose of optimizing the code set, thereby achieving the technical effect of reducing the debugging cost of code debugging on the computing platform and improving the debugging efficiency of code debugging on the computing platform, thereby solving the technical problem of low efficiency of code optimization existing in the related art.

[0119] As an optional solution, the first code set is scanned to obtain a target scanning result, including:

[0120] In a case where a first code subset and a second code subset exist in a first code set, a target scan result is generated, wherein the target scan result is used to indicate that the first code subset to be optimized exists in the first code set, the first code subset is used to sequentially execute a first group of operation operations, the second code subset is used to sequentially execute a second group of operation operations, a first part of the operation operations in the first group of operation operations is the same as a second part of the operation operations in the second group of operation operations, and the second code subset is executed earlier than the first code subset in the first code set;

[0121] The first part of the operation operations includes a plurality of consecutive operation operations starting from the first operation operation in the first group of operation operations, and the second part of the operation operations includes a plurality of consecutive operation operations starting from the first operation operation in the second group of operation operations; or, the first part of the operation operations includes the first operation operation in the first group of operation operations, and the second part of the operation operations includes the first operation operation in the second group of operation operations.

[0122] Optionally, in this embodiment, the above-mentioned first group of operations and the above-mentioned second group of operations include but are not limited to being implemented in the order of the codes in the above-mentioned code set. The operations in the above-mentioned first group of operations and the above-mentioned second group of operations are all performed by calling functions corresponding to the codes in the target computing platform. In this embodiment, the called functions are collectively referred to as Function (Function).

[0123] Figure 7 Schematic diagram of another code debugging method according to an embodiment of the present invention. Figure 7 The steps specifically include but are not limited to the following:

[0124] S1, the client displays the above-mentioned first code set and the interactive object "Scan" for scanning, and also displays the above-mentioned first code subset, including: "Fun1→Fun2→Fun3→Fun6", that is, the above-mentioned first group of operation operations, and the above-mentioned target code subset, including "Fun1→Fun2→Fun3→Fun4", that is, the above-mentioned second group of operation operations.

[0125] S2, “scan” the interactive object to perform the interactive operation;

[0126] S3, in response to the above interaction operation, scanning the above first code set using a code scanning module;

[0127] S4, after the scan is completed, the target scan result is generated and displayed in the client.

[0128] The target scanning result is used to indicate that there is a first code subset to be optimized in the first code set. The target scanning result may include but is not limited to identifying the position of the first code subset to be optimized in the first code set and the reason for optimization, for example, Figure 7 The sign displayed in the "Duplicate calculation! Do you want to optimize?"

[0129] Optionally, in this embodiment, the first part of the operation operations includes a plurality of consecutive operation operations starting from the first operation in the first group of operation operations. Figure 7For example, Fun1→Fun2→Fun3, the second part of the operation includes a plurality of consecutive operations starting from the first operation in the second group of operations. Figure 7 For example, including Fun1→Fun2→Fun3; or, the first part of the operation operation includes the first operation operation in the first group of operation operations, Figure 6 For example, the first code subset consists of val rdd1 = sc.textFile("hello.txt") and rdd1.map(...), then Figure 6 The first code subset shown in the figure includes Fun1: val rdd1 = sc.textFile("hello.txt"), Fun2: rdd1.map(...), then the first part of the operation includes the first operation in the first group of operations, that is, val rdd1 = sc.textFile("hello.txt"), the second part of the operation includes the first operation in the second group of operations, and Figure 6 For example, the target code subset consists of val rdd2 = sc.textFile ("hello.txt") and rdd2.reduce (...), then Figure 6 The first code subset shown in includes Fun1: val rdd2 = sc.textFile("hello.txt"), Fun2: rdd2.reduce(...), then val rdd2 = sc.textFile("hello.txt"), then the above-mentioned second part of the operation operation includes the first operation operation in the second group of operation operations, namely val rdd2 = sc.textFile("hello.txt").

[0130] Optionally, in this embodiment, the third target code includes but is not limited to: Figure 7 The "cache" code shown, the fourth target code is not limited to the following Figure 7 The "Fun5" code shown, the above-mentioned third target code and the above-mentioned fourth target code together constitute the above-mentioned second target code.

[0131] The above is only an example and this embodiment does not impose any specific limitation.

[0132] According to this embodiment, a target scanning result is generated when a first code subset and a second code subset exist in a first code set. The target scanning result includes a target scanning result, which is used to indicate that a first code subset to be optimized exists in the first code set, the first code subset is used to sequentially execute a first group of operations, the second code subset is used to sequentially execute a second group of operations, a first portion of the operations in the first group of operations and a second portion of the operations in the second group of operations are the same, and the second code subset is executed earlier than the first code subset in the first code set. By scanning the first code subset, it is determined that some operations of the first code subset and the second code subset are the same. Furthermore, it is determined that operators in the first code set are repeatedly executed, so that the first code set can be subsequently optimized into the second code set, thereby avoiding the technical problem that repeated execution of operators contained in the code cannot be optimized due to manual code optimization, thereby achieving the purpose of optimizing the code set, thereby achieving the technical effect of reducing the debugging cost and improving the debugging efficiency of code debugging on the computing platform, and thus solving the technical problem of low code optimization efficiency in the related art.

[0133] As an optional solution, optimizing the first code subset in the first code set into a target code subset to obtain a second code set includes:

[0134] The code corresponding to the first part of the operation is deleted in the first code subset, and the code corresponding to the next operation of the first part of the operation is modified to a second target code, wherein the second target code is used to obtain the operation results of the second group of operation operations and perform the next operation on the obtained operation results.

[0135] Optionally, in this embodiment, the above-mentioned deletion of the code corresponding to the first part of the operation in the first code subset may include but is not limited to removing the first part of the operation from the code logic of the first code set, and the above-mentioned modification of the code corresponding to the next operation of the first part of the operation to the second target code may include but is not limited to configuring the above-mentioned second target code to cache the data of the last operation of the above-mentioned first part of the operation, so that the second group of operation operations can directly call the data of the last operation of the above-mentioned first part of the operation for operation.

[0136] Optionally, in this embodiment, the next operation is the operation after the second part of the operation in the second group of operations. Figure 7For example, Fun1→Fun2→Fun3 is the first part of the operation, and Fun5 is the next operation of the first part of the operation. By modifying the Fun5 code to the third target code and adding the fourth target code corresponding to Fun5 after the third target code, Fun1→Fun2→Fun3→Fun5 in the first code subset is modified to cache (corresponding to the aforementioned third target code)→Fun5 (fourth target code).

[0137] Through this embodiment, the code corresponding to the first part of the operation is deleted in the first code subset, and the code corresponding to the next operation of the first part of the operation is modified to the second target code, wherein the second target code is used to obtain the operation results of the second group of operation operations and perform the next operation on the obtained operation results. By deleting the code corresponding to the first part of the operation and modifying the code corresponding to the next operation of the first part of the operation to the second target code, the code logic corresponding to the first code subset is modified from Fun1→Fun2→Fun3→Fun5 to cache→Fun5, so as to avoid the technical problem of repeated execution of operators, thereby achieving the purpose of optimizing the code set, thereby achieving the technical effect of reducing the debugging cost of code debugging on the computing platform and improving the debugging efficiency of code debugging on the computing platform, thereby solving the technical problem of low efficiency of code optimization existing in the related art.

[0138] As an optional solution, the code corresponding to the next operation of the first part of the operation is modified into the second target code, including:

[0139] A third target code is added to the second code subset, and the code corresponding to the next operation of the second part of the operation is modified to a fourth target code, wherein the third target code is set to be executed after the code corresponding to the second part of the operation and before the fourth target code, the third target code is used to store the operation results of the second group of operation operations in the target cache, the fourth target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the second part of the operation on the obtained operation results of the second group of operation operations, and the second target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the first part of the operation on the obtained operation results.

[0140] Optionally, in this embodiment, Figure 8 Schematic diagram of another code debugging method according to an embodiment of the present invention. Figure 8 As shown, the above code debugging may include but is not limited to the following:

[0141] Figure 8 The Job1 shown in the figure is a task instructed to be executed in the above-mentioned first code set. When it is scanned that both Job1 and Job2 need to load data into Rdd1 and Rdd2 to execute fun1 and fun2 operations, a Cache Block is added to the code subset corresponding to Job1 to cache the operation result of fun2. The above-mentioned caching of the operation result of fun2 to the CacheBlock is the above-mentioned third target code. The above-mentioned fourth target code is added to the code subset corresponding to Job2 to extract the operation result of fun2 from the cache.

[0142] It should be noted that during data processing on the target computing platform, the above-mentioned operation process may include but is not limited to cyclic use, that is, after the running code set is determined, the above-mentioned code logic will be executed each time the data to be processed is pulled, thereby avoiding the technical problem of fun1 and fun2 repeatedly creating Rdd1 and Rdd2.

[0143] The above is only an example, and this embodiment does not impose any specific limitation.

[0144] Through this embodiment, a third target code is added to the second code subset, and the code corresponding to the next operation of the second part of the operation operation is modified to a fourth target code, wherein the third target code is set to be executed after the code corresponding to the second part of the operation operation and before the fourth target code. The third target code is used to store the operation results of the second group of operation operations in the target cache, and the fourth target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the second part of the operation operation on the obtained operation results of the second group of operation operations. The second target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the operation on the obtained operation results. In a manner of performing the next operation of the first part of the operation, the code corresponding to the first part of the operation is deleted, and the code corresponding to the next operation of the first part of the operation is modified to the second target code, thereby realizing that the code logic corresponding to the first code subset is modified from Fun1→Fun2→Fun5 to cache→Fun5, so as to avoid the technical problem of repeated creation of Rdd caused by repeated execution of the operator, thereby achieving the purpose of optimizing the code set, thereby achieving the technical effect of reducing the debugging cost of code debugging on the computing platform and improving the debugging efficiency of code debugging on the computing platform, thereby solving the technical problem of low efficiency of code optimization existing in the related art.

[0145] As an optional solution, the first code set is scanned to obtain a target scanning result, including:

[0146] When a first code subset exists in a first code set, a target scan result is generated, wherein the target scan result is used to indicate that a first code subset to be optimized exists in the first code set, and the first code subset is used to transmit data in a first data group to a target data processing node, and the first data group is allowed to include multiple identical data.

[0147] Optionally, in this embodiment, the above-mentioned first code subset uses a predetermined type of operator to realize the transmission of data in the first data group to the target data processing node, which may include but is not limited to configuring the predetermined type of operator as a groupByKey operator, a reduceByKey operator, and an aggregateByKey operator, etc., and the above-mentioned groupByKey operator is not allowed to include multiple identical data, but transmits data one by one each time the data is transmitted. The above-mentioned reduceByKey operator and aggregateByKey operator perform aggregation operations in advance to aggregate the identical data into one piece of data, and can realize the transmission of multiple first data groups of the identical data to the corresponding target data processing node through one transmission.

[0148] Optionally, in this embodiment, the above-mentioned first code subset uses a predetermined type of operator to realize the transmission of data in the first data group to the target data processing node, which may include but is not limited to configuring the predetermined type of operator as a foreach operator, a foreachPartitions operator, etc., wherein the foreach operator executes the processing of one piece of data at a time, and foreachPartitions executes the processing of one Partition at a time (Partition is a node used to store data to be processed).

[0149] Optionally, in this embodiment, the above-mentioned first code subset uses a predetermined type of operator to realize the transmission of data in the first data group to the target data processing node, which may include but is not limited to configuring the predetermined type of operator as a map operator, a mapPartitions operator, etc., wherein the map operator executes the processing of one piece of data at a time, and the mapPartitions executes the processing of one Partition at a time (Partition is a node used to store data to be processed).

[0150] Optionally, in this embodiment, Figure 9 Schematic diagram of another code debugging method according to an embodiment of the present invention. Figure 9 The steps specifically include but are not limited to the following:

[0151] S1, the client displays the first code set and the interactive object "Scan" for scanning, and also displays the first code subset, including the first code subset operated using the groupByKey operator;

[0152] S2, “scan” the interactive object to perform the interactive operation;

[0153] S3, in response to the above interactive operation, scanning the above first code set using a first target code scanning module;

[0154] S4, after the scan is completed, the target scan result is generated and displayed in the client.

[0155] The target scanning result is used to indicate that there is a first code subset to be optimized in the first code set. The target scanning result may include but is not limited to identifying the position of the first code subset to be optimized in the first code set and the reason for optimization, for example, Figure 7 The "Is it optimized?" indicator is displayed.

[0156] Through this embodiment, when a first code subset exists in a first code set, a target scanning result is generated, wherein the target scanning result is used to indicate that a first code subset to be optimized exists in the first code set, and the first code subset is used to transmit the data in the first data group to the target data processing node. The first data group is allowed to include multiple identical numbers. By scanning the first code subset, a low-performance code of the operator corresponding to the code for transmitting the data in the first data group to the target data processing node is scanned. Since the code can only process data one by one, the purpose of optimizing the code set is achieved by modifying the code logic, thereby achieving the technical effect of reducing the debugging cost of code debugging on the computing platform and improving the debugging efficiency of code debugging on the computing platform, thereby solving the technical problem of low efficiency of code optimization existing in the related art.

[0157] As an optional solution, optimizing the first code subset in the first code set into a target code subset to obtain a second code set includes:

[0158] The first code subset is replaced with the target code subset in the first code set. The target code subset is used to aggregate the multiple identical data in the first data packet into one data when the first data packet includes multiple identical data, obtain a second data packet, and transmit the second data packet to the target node.

[0159] Optionally, in this embodiment, Figure 9 For example, it may include but is not limited to the following:

[0160] S1, the first code set and the interactive object "Optimization" for scanning are displayed on the client;

[0161] S2, performs interactive operations on the interactive object “optimize”;

[0162] S3: In response to the above interactive operation, the first code subset is replaced with the target code subset in the first code set.

[0163] By replacing the first code subset operated by the groupByKey operator with the target code subset operated by the reduceByKey operator, when the first data group includes multiple identical data, the multiple identical data in the first data group are aggregated into one piece of data to obtain a second data group, and the second data group is transmitted to the target node.

[0164] It should be noted that the target code subset is used to aggregate the multiple identical data in the first data packet into one data packet when the first data packet includes multiple identical data in the following manner to obtain a second data packet, and transmit the second data packet to the target node:

[0165] Figure 10 Schematic diagram of another code debugging method according to an embodiment of the present invention. Figure 10 As shown, for example, if (hello, 1) appears twice in the first data partition, if the first code subset is used, (hello, 1) will be transmitted twice. After replacing the first code subset with the target code subset, the two (hello, 1) data will be pre-aggregated into one (hello, 2) data, and then (hello, 2) will be transmitted once. The reduceByKey operator will pre-aggregate the data with the same key locally, and then in the shuffle phase, other nodes can reduce the amount of data pulled when pulling data to transmit data, thereby reducing the performance consumption caused by network IO. The groupByKey operator does not have a pre-aggregation operation, and the full amount of data is transmitted and distributed among the nodes in the cluster, which increases too much network data transmission delay. Among them, shuffle is a data interaction operation, and task represents the task performed by the code set.

[0166] The above is only an example, and this embodiment does not impose any specific limitation.

[0167] Through this embodiment, the first code subset is replaced with the target code subset in the first code set. The target code subset is used to aggregate the multiple identical data in the first data group into one data when the first data group includes multiple identical data, obtain the second data group, and transmit the second data group to the target node. By aggregating the multiple identical data in the first data group into one data, obtaining the second data group, and transmitting the second data group to the target node, the technical problem of pulling or transmitting the same data multiple times is avoided, thereby achieving the purpose of optimizing the code set, thereby achieving the technical effect of reducing the debugging cost of code debugging on the computing platform and improving the debugging efficiency of code debugging on the computing platform, thereby solving the technical problem of low efficiency of code optimization existing in the related art.

[0168] As an optional solution, the method further includes at least one of the following:

[0169] If the target scanning result indicates that the first code subset to be optimized exists in the first code set, generating first prompt information, wherein the first prompt information is used to indicate a position of the first code subset in the first code set;

[0170] When the target scanning result indicates that a first code subset to be optimized exists in the first code set, second prompt information is generated, wherein the second prompt information is used to prompt whether the first code subset needs to be replaced with the target code subset.

[0171] Optionally, in this embodiment, the first prompt information and the second prompt information can be pre-configured by the staff, for example, Figure 7 For example, the first code subset is marked in a dotted frame to indicate the location of the first code subset, and “Duplicate! Optimize?” is displayed to indicate whether the first code subset needs to be replaced with the target code subset.

[0172] Optionally, in this embodiment, different color markings may be used as the first prompt information or the second prompt information, including but not limited to.

[0173] The above is only an example, and this embodiment does not impose any specific limitation.

[0174] Through this embodiment, when the target scan result indicates that the first code subset to be optimized exists in the first code set, first prompt information is generated, wherein the first prompt information is used to indicate the position of the first code subset in the first code set. When the target scan result indicates that the first code subset to be optimized exists in the first code set, second prompt information is generated, wherein the second prompt information is used to indicate whether the first code subset needs to be optimized into the second code subset. By generating prompt information for indicating the position of the first code subset in the first code set or indicating whether the first code subset needs to be replaced with the target code subset, the purpose of optimizing the code set is achieved, thereby achieving the technical effect of reducing the debugging cost of code debugging on the computing platform and improving the debugging efficiency of code debugging on the computing platform, thereby solving the technical problem of low efficiency of code optimization existing in the related art.

[0175] As an optional solution, when the target scanning result indicates that a first code subset to be optimized exists in the first code set, optimizing the first code subset in the first code set into a target code subset to obtain a second code set includes:

[0176] If the target scanning result indicates that a first code subset to be optimized exists in the first code set, generating a target optimization solution, wherein the target optimization solution is used to replace the first code subset with a second code subset;

[0177] Obtaining a target execution instruction, wherein the target execution instruction is used to instruct execution of a target optimization solution;

[0178] In response to the target execution instruction, a target optimization solution is executed to replace a first code subset in the first code set with a second code subset to obtain a second code set.

[0179] Optionally, in this embodiment, the above-mentioned target optimization scheme may include but is not limited to providing optimization schemes for optimizing different types of first code subsets to be optimized into target code subsets, for example, the above-mentioned scheme for avoiding repeated calculation of operators, the above-mentioned scheme for avoiding poor operator performance, the above-mentioned scheme for avoiding repeated creation of Rdd, etc.

[0180] Optionally, in this embodiment, the target execution instruction may include but is not limited to the following: Figure 6 The interactive operations shown are performed on the "optimized" interactive objects, for example, Figure 6 In the illustrated scenario, when the "Optimize" button is clicked, the first code subset in the first code set starts to be optimized into a target code subset.

[0181] Optionally, in this embodiment, the reason why the first code subset is the first code subset may be displayed in the display interface of the client, for example, the operator performance is poor, Rdd is repeatedly created, etc.

[0182] The above is only an example, and this embodiment does not impose any specific limitation.

[0183] According to this embodiment, when the target scanning result indicates that the first code subset to be optimized exists in the first code set, a target optimization solution is generated, wherein the target optimization solution is used to replace the first code subset with the second code subset;

[0184] Obtaining a target execution instruction, wherein the target execution instruction is used to instruct execution of a target optimization solution;

[0185] In response to a target execution instruction, a target optimization scheme is executed to replace a first code subset in a first code set with a second code subset to obtain a second code set. By generating a target optimization scheme and obtaining an execution instruction to optimize the first code subset in the first code set, an optimized second code set is obtained, thereby achieving the purpose of optimizing the code set, thereby achieving the technical effect of reducing the debugging cost of code debugging on a computing platform and improving the debugging efficiency of code debugging on a computing platform, thereby solving the technical problem of low efficiency of code optimization existing in related technologies.

[0186] As an optional solution, running a second code set in the target computing platform includes:

[0187] When the second code set is completed in the target computing platform, an operation log of the second code set is generated, wherein the operation log records at least one of the following: the operation parameters of the target computing platform during the process of running the second code set in the target computing platform, and the interaction parameters of the target data operated by the second code set in the process of running the second code set in the target computing platform.

[0188] Optionally, in this embodiment, the second code set may be executed in the target computing platform by including but not limited to obtaining a submission operation instruction. The submission operation instruction may include but is not limited to configuring an interactive object for submitting the code set in the display interface of the client. After obtaining the submission operation performed on the interactive object, the second code set is submitted to the target computing platform in response to the submission operation instruction to execute the second code set in the target computing platform.

[0189] Optionally, in this embodiment, during the process of the target computing platform running the second code set, the operating parameters of the target computing platform can also be obtained in real time.

[0190] Figure 11 FIG. 1 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention. Figure 11 As shown, the operating parameters of the target computing platform when the second code set is running in the target computing platform can be obtained through the visual interface of the operating parameters.

[0191] The above operating parameters may include but are not limited to CPU usage, memory usage, computing time, and whether abnormal events occur.

[0192] Optionally, in this embodiment, the above operation log records the following Figure 11 The operating parameters shown may also include but are not limited to recording Figure 11 The target data interaction parameters shown may include, but are not limited to, CPU usage, memory usage, computation time, and whether an abnormal event has occurred. The target data interaction parameters may include, but are not limited to, transmission path information of the target data, for example, recording transmission path information of the target data to be processed obtained by the data processing node.

[0193] The above is only an example, and this embodiment does not impose any specific limitation.

[0194] Through this embodiment, when the second code set is completed in the target computing platform, an operation log of the second code set is generated, wherein the operation log records at least one of the following: the operation parameters of the target computing platform during the operation of the second code set in the target computing platform, and the interaction parameters of the target data operated by the second code set during the operation of the second code set in the target computing platform. By obtaining the operation parameters of the target computing platform when running the second code set and the interaction parameters of the target data in real time, the second code set can be readjusted subsequently, thereby achieving the purpose of further optimizing the code set, thereby achieving the technical effect of reducing the debugging cost of code debugging on the computing platform and improving the debugging efficiency of code debugging on the computing platform, thereby solving the technical problem of low efficiency of code optimization existing in related technologies.

[0195] As an optional solution, after generating the execution log of the second code set, the method further includes at least one of the following:

[0196] In a case where the loading duration indicated by the running parameter is greater than or equal to a preset duration threshold, a first section of code is added to the second code set, wherein the loading duration is the duration for loading target data when the target computing platform runs the second code set, and the first section of code is used to increase the capacity of each partition in the partition set of the target computing platform, or to increase the capacity of each partition in the partition set of the target computing platform and reduce the number of partitions in the partition set, and the target computing platform is used to load the target data from the partition set;

[0197] In a case where the value of the memory occupancy parameter represented by the running parameter is greater than or equal to a preset occupancy threshold, adding a second segment of code to the second code set, wherein the value of the memory occupancy parameter is used to represent the memory occupied when the target computing platform runs the second code set to process the target data, and the second segment of code is used to increase the memory allowed to be used by the target computing platform;

[0198] When the second code set includes codes for performing a transmission operation or a collection operation on part or all of the target data, a third code segment is added to the second code set, wherein the third code segment is used to increase the memory allowed to be used by the target computing platform.

[0199] Optionally, in this embodiment, the above-mentioned loading duration is the duration for loading target data when the target computing platform runs the second code set. The target computing platform pulls data from each partition in the partition set to process the data. When the above-mentioned loading duration exceeds the preset duration threshold, it is considered that the current loading has timed out, the amount of data to be loaded is too large or there are too many small files. Therefore, the performance is optimized by increasing the capacity of each partition in the partition set of the target computing platform, or by increasing the capacity of each partition in the partition set of the target computing platform and reducing the number of partitions in the partition set. For example, when loading the data source is slow, it means that there are too many small files of the source data. The capacity of the partition partition (partition in the partition set) can be increased, and the number of partitions can be reduced by using coalesce, thereby reducing the number of small files to improve the processing speed of the target computing platform, and then shortening the time for loading data so that the time for loading data can meet business needs.

[0200] Optionally, in this embodiment, when the memory occupied by the target computing platform when running the second code set to process target data is greater than or equal to a preset occupancy threshold, a second segment of code is added to the second code set to increase the memory allowed to be used by the target computing platform.

[0201] For example, during the execution of the second code set running on the target computing platform, memory overflow occurs or the memory occupancy rate is 100% for a long time (corresponding to the aforementioned memory occupied by the second code set running on the target computing platform to process target data being greater than or equal to the preset occupancy threshold), then it is necessary to increase the executor-memory parameter so that the operating efficiency of the target computing platform can meet business needs.

[0202] Optionally, in this embodiment, when the second code set includes a code subset for performing a transmission operation or a collection operation on part or all of the target data, a code subset for increasing the memory allowed to be used by the target computing platform is added to the second code set.

[0203] For example, during the execution of the second code set on the target computing platform, operations such as broadcast and collect occur. By appropriately increasing the driver-memory parameter, the operating efficiency of the target computing platform can meet business needs.

[0204] Figure 12 FIG. 1 is a schematic diagram of another optional code debugging method according to an embodiment of the present invention. Figure 12 The specific contents include but are not limited to the following:

[0205] The client display interface 1202 is used to indicate a situation where the value of the memory occupancy parameter is greater than or equal to a preset occupancy threshold, wherein the dotted portion is the preset occupancy threshold, and the shaded portion is the memory occupancy parameter of the target computing platform when running the second code set. When the shaded portion covers the dotted portion, it is determined that the value of the above memory occupancy parameter is greater than or equal to the preset occupancy threshold.

[0206] The client display interface 1204 is used to indicate a situation where the loading time exceeds a preset time threshold. Assuming that the preset time threshold is 60 minutes, when the loading time displayed on the above-mentioned client display interface 1204 exceeds 50 minutes, which is 1 hour, 23 minutes and 25 seconds, it is determined that the loading time is greater than or equal to the preset time threshold.

[0207] The client display interface 1206 is used to indicate whether an abnormal event has occurred. For example, when a collection or transmission operation is found, the representation used to indicate the abnormal event changes from a blank space to a shaded portion, and it is determined that the second code set includes a fifth code subset for performing a transmission operation or a collection operation on part or all of the target data.

[0208] The above is only an example, and this embodiment does not impose any specific limitation.

[0209] Through this embodiment, when the loading time represented by the running parameter is greater than or equal to the preset time threshold, a first section of code is added to the second code set, wherein the loading time is the time for loading the target data when the target computing platform runs the second code set, and the first section of code is used to increase the capacity of each partition in the partition set of the target computing platform, or to increase the capacity of each partition in the partition set of the target computing platform and reduce the number of partitions in the partition set, and the target computing platform is used to load the target data from the partition set. When the value of the memory occupancy parameter represented by the running parameter is greater than or equal to the preset occupancy threshold, a second section of code is added to the second code set, wherein the value of the memory occupancy parameter is used to indicate that the target computing platform runs the second The memory occupied by the code set when processing the target data, the second code segment is used to increase the memory allowed to be used by the target computing platform. When the second code set includes code for performing transmission operations or collection operations on part or all of the target data, a third code segment is added to the second code set, wherein the third code segment is used to increase the memory allowed to be used by the target computing platform. By outputting code debugging suggestions according to the operating parameters to optimize the above-mentioned second code set, the purpose of further optimizing the code set can be achieved, thereby achieving the technical effect of reducing the debugging cost of code debugging on the computing platform and improving the debugging efficiency of code debugging on the computing platform, thereby solving the technical problem of low efficiency of code optimization existing in the relevant technology.

[0210] As an optional solution, after generating the execution log of the second code set, the method further includes:

[0211] If the running parameter indicates that data skew occurs during the running of the second code set, determining whether the second code set includes code for performing a join operation on a table formed by part or all of the target data;

[0212] In a case where the second code set includes code for performing a join operation on a table formed by part or all of the data in the target data, searching the target data for a first data set and a second data set, wherein the magnitude of the data in the first data set is smaller than a preset first data magnitude and the data in the first data set is configured to perform the join operation, the magnitude of the data in the second data set is greater than a preset second data magnitude and the data in the second data set is configured to perform the join operation, and the first data magnitude is smaller than the second data magnitude;

[0213] When the first data set is found, the first data set is broadcasted to the plurality of data processing nodes, so that a join operation is performed on the table formed by the data in the first data set in parallel by the plurality of data processing nodes to obtain a first operation result;

[0214] When the second data set is found, the data in the second data set is split into a first data subset having a data skew indicator and a second data subset not having a data skew indicator;

[0215] Broadcasting the first data subset to multiple data processing nodes, so that the multiple data processing nodes perform a join operation on the table formed by the data in the first data subset in parallel to obtain a second operation result;

[0216] directly performing a join operation on the table formed by the data in the second data subset to obtain a third operation result;

[0217] The second operation result and the third operation result are combined to obtain a fourth operation result.

[0218] Optionally, in this embodiment, the above-mentioned data skew is due to the fact that when shuffle is performed, the same key on each node must be pulled to a task on a certain node for processing, such as aggregation or join operations according to the key. If the amount of data corresponding to a certain key is particularly large, data skew will occur. For example, most keys correspond to 10 data items, but some keys correspond to 1 million data items. Then, most tasks will only be allocated 10 data items, while some tasks may be allocated 1 million data items. Therefore, the running progress of the target computing platform is determined by the task with the longest running time. In other words, when it is found in the interaction parameters and running parameters of the above-mentioned target data that most nodes process quickly, while some nodes process slowly, it can be determined that data skew has occurred during the operation of the second code set.

[0219] Optionally, in this embodiment, the above connection operation may include but is not limited to the join operation included in SQL, which can be roughly divided into the following three categories according to their functions:

[0220] INNER JOIN (inner join, or equijoin): obtains records in two tables that have a join matching relationship.

[0221] LEFT JOIN (left join): Get all records in the left table (table1), without worrying about whether there are corresponding matching records in the right table (table2): records with corresponding connections will be displayed according to the connection, and records without corresponding connections will be filled with null

[0222] RIGHT JOIN: The opposite of LEFT JOIN, it retrieves all records from the right table (table2), even if there are no matching records in the left table (table1).

[0223] The first data set may include but is not limited to the left table.

[0224] Figure 13 Schematic diagram of another code debugging method according to an embodiment of the present invention. Figure 13 The specific steps are as follows:

[0225] S1, check whether the above second code set has a join operation. If the data level of the left table of the join operation is smaller than the first data level, it can include but is not limited to broadcasting to each executor (corresponding to the above data processing node) in the form of broadcast (corresponding to the above broadcast operation), and then use the map method to implement the join function to obtain the above first operation result.

[0226] S2: If the magnitude of the left table is greater than the second data magnitude and cannot be fully stored in memory, consider sampling and splitting the large data into keys with data skew (corresponding to the first data subset mentioned above) and keys without data skew (corresponding to the second data subset mentioned above). For the keys with data skew, use the broadcast+mapjoin method mentioned above to optimize them, and obtain the second operation result mentioned above.

[0227] In S3, keys without data skew are directly joined to obtain the third operation result.

[0228] S4: Perform a union all operation on the split and executed results to obtain the fourth operation result.

[0229] in, Figure 14 Schematic diagram of another code debugging method according to an embodiment of the present invention. Figure 14 As shown, by sampling and splitting Rdd_flow (a table with a magnitude greater than the second data magnitude) or Rdd_black (a table with a magnitude less than the first data magnitude), the normal data obtained by the split, for example, the data in Rdd_flow_normal and Rdd_black_normal, are directly joined to obtain Rdd_ret_normal including normal data, and the abnormal data obtained by the split (for example, a table with data skew), for example, Rdd_flow_skew, is map-operated, and the data in Rdd_black_skew is broadcast-operated to obtain Rdd_ret_skew including abnormal data. Finally, the data in Rdd_ret_normal and Rdd_ret_skew are spliced ​​(union all) to obtain the final Rdd_ret.

[0230] Optionally, in this embodiment, when the second code set does not include code for performing a join operation on a table formed by part or all of the target data, it is determined that the data skew is caused by an aggregation operation. If the data skew problem is caused by an aggregation operation, a random prefix is ​​added to the key of the data skew, and pre-aggregation is performed. The random prefix is ​​then removed, and finally global aggregation is performed to solve the technical problem of data skew that occurs during data processing on the target computing platform in the related technology.

[0231] According to this embodiment, when the operating parameters indicate that data skew occurs during the operation of the second code set, it is determined whether the second code set includes code for performing a join operation on a table formed by part or all of the data in the target data. When the second code set includes code for performing a join operation on a table formed by part or all of the data in the target data, a first data set and a second data set are searched in the target data, wherein the magnitude of the data in the first data set is smaller than a preset first data magnitude, and the data in the first data set is set to perform a join operation, and the magnitude of the data in the second data set is greater than a preset second data magnitude, and the data in the second data set is set to perform a join operation, and the first data magnitude is smaller than the second data magnitude. When the first data set is found, the first data set is broadcast to multiple data processing nodes, so that the join operation is performed on the table formed by the data in the first data set in parallel through the multiple data processing nodes, and the first operation is obtained. As a result, when the second data set is found, the data in the second data set is split into a first data subset with a data skew identifier and a second data subset without a data skew identifier, and the first data subset is broadcast to multiple data processing nodes, so that a join operation is performed on the table formed by the data in the first data subset in parallel through multiple data processing nodes to obtain a second operation result, and a join operation is directly performed on the table formed by the data in the second data subset to obtain a third operation result, and the second operation result and the third operation result are merged to obtain a fourth operation result. By adaptively performing corresponding operations based on whether the above-mentioned second code set contains a join operation and the data scale of different tables after the join operation, the purpose of further optimizing the code set can be achieved, thereby reducing the debugging cost of code debugging on the computing platform, and solving the technical problem of data skew that occurs during data processing on the target computing platform in the related technology.

[0232] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0233] According to another aspect of the embodiment of the present invention, a code debugging device for implementing the above code debugging method is also provided. Figure 15 As shown, the device includes:

[0234] An acquisition module 1502 is configured to acquire a first code set to be executed;

[0235] A scanning module 1504 is configured to scan the first code set to obtain a target scanning result, wherein the target scanning result is used to indicate whether the first code subset to be optimized exists in the first code set;

[0236] an optimization module 1506 configured to, if the target scanning result indicates that a first code subset to be optimized exists in the first code set, replace the first code subset in the first code set with a target code subset to obtain a second code set, wherein the second code set includes the target code subset and code in the first code set other than the first code subset, the first code subset and the target code subset have the same function, and running resources required to run the target code subset are less than running resources required to run the first code subset;

[0237] The running module 1508 is configured to run the second code set in the target computing platform.

[0238] As an optional solution, the scanning module 1504 includes:

[0239] The first scanning module 1504 is used to generate a target scanning result when a first code subset and a second code subset exist in the first code set, wherein the target scanning result is used to indicate that a first code subset to be optimized exists in the first code set, the first code subset is used to load target data into a created first data set, the second code subset is used to load target data into a created second data set, and the second code subset is executed earlier than the first code subset in the first code set.

[0240] As an optional solution, the optimization module 1506 includes:

[0241] The first optimization unit is used to delete the first code in the first code subset and modify the second code into a first target code when the first code subset includes the first code and the second code, wherein the first code is used to load target data into the first data set, the second code is used to perform a target operation on the target data in the first data set, and the first target code is used to perform the target operation on the target data in the second data set.

[0242] As an optional solution, the scanning module 1504 includes:

[0243] The second scanning module 1504 is used to generate a target scanning result when a first code subset and a second code subset exist in the first code set, wherein the target scanning result is used to indicate that a first code subset to be optimized exists in the first code set, the first code subset is used to sequentially execute a first group of operations, the second code subset is used to sequentially execute a second group of operations, a first part of the operations in the first group of operations and a second part of the operations in the second group of operations are the same, and the second code subset is executed earlier than the first code subset in the first code set; wherein the first part of the operations includes a plurality of consecutive operations starting from the first operation in the first group of operations, and the second part of the operations includes a plurality of consecutive operations starting from the first operation in the second group of operations; or, the first part of the operations includes the first operation in the first group of operations, and the second part of the operations includes the first operation in the second group of operations.

[0244] As an optional solution, the optimization module 1506 includes:

[0245] The second optimization unit is used to delete the code corresponding to the first part of the operation in the first code subset, and modify the code corresponding to the next operation of the first part of the operation to a second target code, wherein the second target code is used to obtain the operation results of the second group of operation operations and perform the next operation on the obtained operation results.

[0246] As an optional solution, the optimization module 1506 includes:

[0247] The third optimization unit is used to add a third target code to the second code subset, and modify the code corresponding to the next operation of the second part of the operation to a fourth target code, wherein the third target code is set to be executed after the code corresponding to the second part of the operation and before the fourth target code, the third target code is used to store the operation results of the second group of operation operations in the target cache, the fourth target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the second part of the operation on the obtained operation results of the second group of operation operations, and the second target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the first part of the operation on the obtained operation results.

[0248] As an optional solution, the scanning module 1504 includes:

[0249] The third scanning module 1504 is used to generate a target scanning result when the first code subset exists in the first code set, wherein the target scanning result is used to indicate that the first code subset to be optimized exists in the first code set, and the first code subset is used to transmit data in the first data group to the target data processing node, and the first data group is allowed to include multiple identical data.

[0250] As an optional solution, the optimization module 1506 includes:

[0251] The fourth optimization unit is used to replace the first code subset with the target code subset in the first code set. The target code subset is used to aggregate the multiple identical data in the first data group into one data when the first data group includes multiple identical data, obtain the second data group, and transmit the second data group to the target node.

[0252] As an optional solution, the device further includes at least one of the following:

[0253] a first prompting module, configured to generate first prompt information when the target scanning result indicates that a first code subset to be optimized exists in the first code set, wherein the first prompt information is used to indicate a position of the first code subset in the first code set;

[0254] The second prompt module is configured to generate second prompt information when the target scanning result indicates that a first code subset to be optimized exists in the first code set, wherein the second prompt information is used to prompt whether the first code subset needs to be replaced with the target code subset.

[0255] As an optional solution, the optimization module 1506 includes:

[0256] a generating unit, configured to generate a target optimization solution when the target scanning result indicates that a first code subset to be optimized exists in the first code set, wherein the target optimization solution is used to replace the first code subset with a second code subset;

[0257] an acquisition unit, configured to acquire a target execution instruction, wherein the target execution instruction is used to instruct execution of a target optimization solution;

[0258] The execution unit is configured to execute a target optimization solution in response to a target execution instruction, so as to replace a first code subset in the first code set with a second code subset to obtain a second code set.

[0259] As an optional solution, the device is also used to:

[0260] After running the second code set in the target computing platform, when the second code set completes running on the target computing platform, an operation log of the second code set is generated, wherein the operation log records at least one of the following: operating parameters of the target computing platform during the process of running the second code set in the target computing platform, and interaction parameters of the target data operated by the second code set during the process of running the second code set in the target computing platform.

[0261] As an optional solution, after generating the operation log of the second code set, the apparatus further includes at least one of the following:

[0262] a first processing module, configured to add a first segment of code to a second code set when a loading duration indicated by an operating parameter is greater than or equal to a preset duration threshold, wherein the loading duration is a duration for loading target data when a target computing platform runs the second code set, the first segment of code being configured to increase a capacity of each partition in a partition set of the target computing platform, or to increase a capacity of each partition in a partition set of the target computing platform and to reduce a number of partitions in the partition set, the target computing platform being configured to load target data from the partition set;

[0263] a second processing module, configured to add a second segment of code to the second code set when the value of the memory occupancy parameter represented by the operating parameter is greater than or equal to a preset occupancy threshold, wherein the value of the memory occupancy parameter is used to represent the memory occupied by the target computing platform when the target computing platform runs the second code set to process the target data, and the second segment of code is used to increase the memory allowed to be used by the target computing platform;

[0264] The third processing module is used to add a third segment of code to the second code set when the second code set includes code for performing transmission operations or collection operations on part or all of the target data, wherein the third segment of code is used to increase the memory allowed to be used by the target computing platform.

[0265] As an optional solution, the device is also used to:

[0266] If the running parameter indicates that data skew occurs during the running of the second code set, determining whether the second code set includes code for performing a join operation on a table formed by part or all of the target data;

[0267] In a case where the second code set includes code for performing a join operation on a table formed by part or all of the data in the target data, searching the target data for a first data set and a second data set, wherein the magnitude of the data in the first data set is smaller than a preset first data magnitude and the data in the first data set is configured to perform the join operation, the magnitude of the data in the second data set is greater than a preset second data magnitude and the data in the second data set is configured to perform the join operation, and the first data magnitude is smaller than the second data magnitude;

[0268] When the first data set is found, the first data set is broadcasted to the plurality of data processing nodes, so that a join operation is performed on the table formed by the data in the first data set in parallel by the plurality of data processing nodes to obtain a first operation result;

[0269] When the second data set is found, the data in the second data set is split into a first data subset having a data skew indicator and a second data subset not having a data skew indicator;

[0270] Broadcasting the first data subset to multiple data processing nodes, so that the multiple data processing nodes perform a join operation on the table formed by the data in the first data subset in parallel to obtain a second operation result;

[0271] directly performing a join operation on the table formed by the data in the second data subset to obtain a third operation result;

[0272] The second operation result and the third operation result are combined to obtain a fourth operation result.

[0273] According to another aspect of the embodiment of the present invention, an electronic device for implementing the above code debugging method is also provided. The electronic device may be Figure 1 The terminal device or server shown in FIG. This embodiment is described by taking the electronic device as a server as an example. Figure 16 As shown, the electronic device includes a memory 1602 and a processor 1604. The memory 1602 stores a computer program, and the processor 1604 is configured to execute the steps in any of the above method embodiments through the computer program.

[0274] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0275] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0276] S1, obtaining a first code set to be executed;

[0277] S2, scanning the first code set to obtain a target scanning result, wherein the target scanning result is used to indicate whether a first code subset to be optimized exists in the first code set;

[0278] S3, if the target scanning result indicates that the first code subset to be optimized exists in the first code set, replacing the first code subset in the first code set with the target code subset to obtain a second code set, wherein the second code set includes the target code subset and codes in the first code set other than the first code subset, the first code subset and the target code subset have the same function, and running resources required to run the target code subset are less than running resources required to run the first code subset;

[0279] S4, running the second code set in the target computing platform.

[0280] Alternatively, those skilled in the art will appreciate that Figure 16 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 16 It does not limit the structure of the electronic device. For example, the electronic device may also include Figure 16 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 16 Different configurations shown.

[0281] Among them, the memory 1602 can be used to store software programs and modules, such as the program instructions / modules corresponding to the code debugging method and device in the embodiment of the present invention. The processor 1604 executes various functional applications and data processing by running the software programs and modules stored in the memory 1602, that is, realizing the above-mentioned code debugging method. The memory 1602 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1602 may further include a memory remotely located relative to the processor 1604, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1602 can be used for information such as target data or code sets, but is not limited to it. As an example, if Figure 16 As shown, the above-mentioned memory 1602 may include, but is not limited to, the acquisition module 1502, scanning module 1504, optimization module 1506 and running module 1508 in the above-mentioned code debugging method device. In addition, it may also include, but is not limited to, other module units in the above-mentioned code debugging method device, which will not be repeated in this example.

[0282] Optionally, the transmission device 1606 is configured to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one embodiment, the transmission device 1606 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 1606 is a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0283] In addition, the electronic device further includes: a display 1608 for displaying the prompt information or optimization solution; and a connection bus 1610 for connecting various module components in the electronic device.

[0284] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting multiple nodes through network communication. The nodes may form a peer-to-peer (P2P) network, and any computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0285] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the aforementioned code debugging method. The computer program is configured to execute the steps of any of the aforementioned method embodiments when executed.

[0286] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0287] S1, obtaining a first code set to be executed;

[0288] S2, scanning the first code set to obtain a target scanning result, wherein the target scanning result is used to indicate whether a first code subset to be optimized exists in the first code set;

[0289] S3, if the target scanning result indicates that the first code subset to be optimized exists in the first code set, replacing the first code subset in the first code set with the target code subset to obtain a second code set, wherein the second code set includes the target code subset and codes in the first code set other than the first code subset, the first code subset and the target code subset have the same function, and running resources required to run the target code subset are less than running resources required to run the first code subset;

[0290] S4, running the second code set in the target computing platform.

[0291] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0292] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0293] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing one or more computer devices (such as personal computers, servers, or network devices) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0294] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0295] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0296] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0297] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0298] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A code debugging method, characterized in that: include: Obtaining a first code set to be executed; Scanning the first code set to obtain a target scanning result that identifies a location of a first code subset to be optimized in the first code set and a reason for the optimization, wherein the target scanning result indicates whether the first code subset to be optimized exists in the first code set; If the target scan result indicates that the first code subset to be optimized exists in the first code set, replacing the first code subset in the first code set with a target code subset to obtain a second code set, wherein the second code set includes the target code subset and codes in the first code set other than the first code subset, the first code subset and the target code subset have the same function, and running resources required to run the target code subset are less than running resources required to run the first code subset; running the second set of codes in a target computing platform; The method also includes: generating the target scan result when the first code subset and the second code subset exist in the first code set, wherein the target scan result is used to indicate that the first code subset to be optimized exists in the first code set, the first code subset is used to load target data into the created first data set, the second code subset is used to load the target data into the created second data set, and the second code subset is executed earlier than the first code subset in the first code set.

2. The method according to claim 1, characterized in that The replacing the first code subset in the first code set with a target code subset to obtain a second code set includes: In the case where the first code subset includes a first code and a second code, the first code is deleted from the first code subset and the second code is modified to a first target code, wherein the first code is used to load the target data into the first data set, the second code is used to perform a target operation on the target data in the first data set, and the first target code is used to perform the target operation on the target data in the second data set.

3. The method according to claim 1, characterized in that The scanning of the first code set to obtain a target scanning result identifying a location of the first code subset to be optimized in the first code set and a reason for the optimization, includes: generating the target scan result when the first code subset and the second code subset exist in the first code set, wherein the target scan result is used to indicate that the first code subset to be optimized exists in the first code set, the first code subset is used to sequentially execute a first group of operation operations, the second code subset is used to sequentially execute a second group of operation operations, a first part of the operation operations in the first group of operation operations and a second part of the operation operations in the second group of operation operations are the same, and the second code subset is executed earlier than the first code subset in the first code set; The first part of the operation operations includes a plurality of consecutive operation operations starting from the first operation operation in the first group of operation operations, and the second part of the operation operations includes a plurality of consecutive operation operations starting from the first operation operation in the second group of operation operations; or, the first part of the operation operations includes the first operation operation in the first group of operation operations, and the second part of the operation operations includes the first operation operation in the second group of operation operations.

4. The method according to claim 3, characterized in that The replacing the first code subset in the first code set with a target code subset to obtain a second code set includes: The code corresponding to the first part of the operation is deleted in the first code subset, and the code corresponding to the next operation of the first part of the operation is modified to a second target code, wherein the second target code is used to obtain the operation result of the second group of operation operations and perform the next operation on the obtained operation result.

5. The method according to claim 4, characterized in that The step of modifying the code corresponding to the next operation of the first part of the operation into the second target code includes: A third target code is added to the second code subset, and the code corresponding to the next operation of the second part of the operation is modified to a fourth target code, wherein the third target code is set to be executed after the code corresponding to the second part of the operation and before the fourth target code, the third target code is used to store the operation results of the second group of operation operations in the target cache, the fourth target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the second part of the operation on the obtained operation results of the second group of operation operations, and the second target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the first part of the operation on the obtained operation results.

6. The method according to claim 1, characterized in that The scanning of the first code set to obtain a target scanning result identifying a location of the first code subset to be optimized in the first code set and a reason for the optimization, includes: When the first code subset exists in the first code set, the target scan result is generated, wherein the target scan result is used to indicate that the first code subset to be optimized exists in the first code set, and the first code subset is used to transmit data in a first data group to a target data processing node, and the first data group is allowed to include multiple identical data.

7. The method according to claim 6, characterized in that The replacing the first code subset in the first code set with a target code subset to obtain a second code set includes: The first code subset is replaced with the target code subset in the first code set, wherein the target code subset is used to aggregate the multiple identical data in the first data packet into one data when the first data packet includes multiple identical data, obtain a second data packet, and transmit the second data packet to a target node.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises at least one of the following: If the target scanning result indicates that the first code subset to be optimized exists in the first code set, generating first prompt information, wherein the first prompt information is used to indicate a position of the first code subset in the first code set; When the target scanning result indicates that the first code subset to be optimized exists in the first code set, second prompt information is generated, wherein the second prompt information is used to prompt whether the first code subset needs to be replaced with the target code subset.

9. The method according to any one of claims 1 to 7, characterized in that When the target scanning result indicates that the first code subset to be optimized exists in the first code set, replacing the first code subset in the first code set with a target code subset to obtain a second code set includes: If the target scanning result indicates that the first code subset to be optimized exists in the first code set, generating a target optimization solution, wherein the target optimization solution is used to replace the first code subset with the target code subset; Obtaining a target execution instruction, wherein the target execution instruction is used to instruct execution of the target optimization solution; In response to the target execution instruction, the target optimization solution is executed to replace the first code subset in the first code set with the target code subset to obtain the second code set.

10. The method according to any one of claims 1 to 7, characterized in that After running the second set of codes in the target computing platform, the method further includes: When the second code set is completed in the target computing platform, an operation log of the second code set is generated, wherein the operation log records at least one of the following: the operation parameters of the target computing platform during the operation of the second code set in the target computing platform, and the interaction parameters of the target data operated by the second code set in the process of running the second code set in the target computing platform.

11. The method according to claim 10, characterized in that After generating the execution log of the second code set, the method further includes at least one of the following: In a case where the loading duration indicated by the running parameter is greater than or equal to a preset duration threshold, a first section of code is added to the second code set, wherein the loading duration is the duration for loading the target data when the target computing platform runs the second code set, and the first section of code is used to increase the capacity of each partition in the partition set of the target computing platform, or to increase the capacity of each partition in the partition set of the target computing platform and reduce the number of partitions in the partition set, and the target computing platform is used to load the target data from the partition set; When the value of the memory occupancy parameter represented by the operating parameter is greater than or equal to a preset occupancy threshold, adding a second segment of code to the second code set, wherein the value of the memory occupancy parameter is used to represent the memory occupied by the target computing platform when the second code set is executed to process the target data, and the second segment of code is used to increase the memory allowed to be used by the target computing platform; When the second code set includes code for performing a transmission operation or a collection operation on part or all of the target data, a third segment of code is added to the second code set, wherein the third segment of code is used to increase the memory allowed to be used by the target computing platform.

12. The method according to claim 10, characterized in that After generating the operation log of the second code set, the method further includes: If the running parameter indicates that data skew occurs during the running of the second code set, determining whether the second code set includes code for performing a join operation on a table formed by part or all of the target data; In a case where the second code set includes the code for performing a join operation on a table formed by part or all of the data in the target data, searching the target data for a first data set and a second data set, wherein the magnitude of the data in the first data set is smaller than a preset first data magnitude and the data in the first data set is configured to perform the join operation, and the magnitude of the data in the second data set is greater than a preset second data magnitude and the data in the second data set is configured to perform the join operation, and the first data magnitude is smaller than the second data magnitude; When the first data set is found, broadcasting the first data set to multiple data processing nodes, so that the multiple data processing nodes perform the join operation on the table formed by the data in the first data set in parallel to obtain a first operation result; In the case where the second data set is found, splitting the data in the second data set into a first data subset having a data skew indicator and a second data subset not having a data skew indicator; Broadcasting the first data subset to the multiple data processing nodes, so that the multiple data processing nodes perform the join operation on the table formed by the data in the first data subset in parallel to obtain a second operation result; directly performing the join operation on the table formed by the data in the second data subset to obtain a third operation result; The second operation result and the third operation result are combined to obtain a fourth operation result.

13. A code debugging device, characterized in that: include: An acquisition module, configured to acquire a first code set to be executed; a scanning module, configured to scan the first code set to obtain a target scanning result that identifies a location of a first code subset to be optimized in the first code set and a reason for the optimization, wherein the target scanning result indicates whether the first code subset to be optimized exists in the first code set; an optimization module, configured to, if the target scanning result indicates that the first code subset to be optimized exists in the first code set, replace the first code subset in the first code set with a target code subset to obtain a second code set, wherein the second code set includes the target code subset and codes in the first code set other than the first code subset, the first code subset and the target code subset have the same function, and running resources required to run the target code subset are less than running resources required to run the first code subset; An execution module, configured to execute the second code set in a target computing platform; The device is further used to: generate the target scan result when the first code subset and the second code subset exist in the first code set, wherein the target scan result is used to indicate that the first code subset to be optimized exists in the first code set, the first code subset is used to load target data into the created first data set, the second code subset is used to load the target data into the created second data set, and the second code subset is executed earlier than the first code subset in the first code set.

14. The device according to claim 13, characterized in that The optimization module includes: A first optimization unit is configured to, when the first code subset includes a first code and a second code, delete the first code from the first code subset and modify the second code into a first target code, wherein the first code is configured to load the target data into the first data set, the second code is configured to perform a target operation on the target data in the first data set, and the first target code is configured to perform the target operation on the target data in the second data set.

15. The device according to claim 13, characterized in that The scanning module includes: a second scanning module, configured to generate the target scanning result when a first code subset and a second code subset exist in the first code set, wherein the target scanning result indicates that the first code subset to be optimized exists in the first code set, the first code subset is configured to sequentially execute a first group of operation operations, the second code subset is configured to sequentially execute a second group of operation operations, a first portion of the operation operations in the first group of operation operations and a second portion of the operation operations in the second group of operation operations are the same, and the second code subset is executed earlier than the first code subset in the first code set; The first part of the operation operations includes a plurality of consecutive operation operations starting from the first operation operation in the first group of operation operations, and the second part of the operation operations includes a plurality of consecutive operation operations starting from the first operation operation in the second group of operation operations; or, the first part of the operation operations includes the first operation operation in the first group of operation operations, and the second part of the operation operations includes the first operation operation in the second group of operation operations.

16. The device according to claim 15, characterized in that The optimization module includes: The second optimization unit is used to delete the code corresponding to the first part of the operation in the first code subset, and modify the code corresponding to the next operation of the first part of the operation to a second target code, wherein the second target code is used to obtain the operation result of the second group of operation operations and perform the next operation on the obtained operation result.

17. The device according to claim 16, characterized in that The optimization module includes: A third optimization unit is used to add a third target code to the second code subset and modify the code corresponding to the next operation of the second part of the operation to a fourth target code, wherein the third target code is set to be executed after the code corresponding to the second part of the operation and before the fourth target code, the third target code is used to store the operation results of the second group of operation operations in the target cache, the fourth target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the second part of the operation on the obtained operation results of the second group of operation operations, and the second target code is used to obtain the operation results of the second group of operation operations from the target cache, and perform the next operation of the first part of the operation on the obtained operation results.

18. The device according to claim 13, characterized in that The scanning module includes: a third scanning module, configured to generate the target scanning result when a first code subset exists in the first code set, wherein the target scanning result is used to indicate that the first code subset to be optimized exists in the first code set, the first code subset is used to transmit data in a first data group to a target data processing node, and the first data group is allowed to include multiple identical data.

19. The device according to claim 18, characterized in that The optimization module includes: A fourth optimization unit is used to replace the first code subset with the target code subset in the first code set, and the target code subset is used to aggregate the multiple identical data in the first data group into one data when the first data group includes multiple identical data, to obtain a second data group, and transmit the second data group to the target node.

20. The device according to any one of claims 13 to 19, characterized in that The device further comprises at least one of the following: a first prompting module, configured to generate first prompt information if the target scanning result indicates that a first code subset to be optimized exists in the first code set, wherein the first prompt information is used to indicate a position of the first code subset in the first code set; A second prompt module is configured to generate second prompt information when the target scanning result indicates that a first code subset to be optimized exists in the first code set, wherein the second prompt information is used to prompt whether the first code subset needs to be replaced with the target code subset.

21. The device according to any one of claims 13 to 19, characterized in that The optimization module includes: a generating unit, configured to generate a target optimization solution when the target scanning result indicates that a first code subset to be optimized exists in the first code set, wherein the target optimization solution is used to replace the first code subset with the second code subset; an acquiring unit, configured to acquire a target execution instruction, wherein the target execution instruction is used to instruct execution of the target optimization solution; An execution unit is configured to execute the target optimization solution in response to the target execution instruction, so as to replace the first code subset in the first code set with the second code subset to obtain the second code set.

22. The device according to any one of claims 13 to 19, characterized in that The device is also used for: After running the second code set in the target computing platform, when the second code set completes running on the target computing platform, an operation log of the second code set is generated, wherein the operation log records at least one of the following: operating parameters of the target computing platform during the process of running the second code set in the target computing platform, and interaction parameters of the target data operated by the second code set during the process of running the second code set in the target computing platform.

23. The device according to claim 22, characterized in that After generating the operation log of the second code set, the apparatus further includes at least one of the following: a first processing module, configured to add a first section of code to the second code set if the loading duration represented by the running parameter is greater than or equal to a preset duration threshold, wherein the loading duration is the duration for loading the target data when the target computing platform runs the second code set, and the first section of code is configured to increase the capacity of each partition in the partition set of the target computing platform, or to increase the capacity of each partition in the partition set of the target computing platform and reduce the number of partitions in the partition set, and the target computing platform is configured to load the target data from the partition set; a second processing module, configured to add a second segment of code to the second code set when a value of a memory occupancy parameter represented by the operating parameter is greater than or equal to a preset occupancy threshold, wherein the value of the memory occupancy parameter is used to represent the memory occupied by the target computing platform when the second code set is executed to process the target data, and the second segment of code is used to increase the memory allowed to be used by the target computing platform; A third processing module is used to add a third segment of code to the second code set when the second code set includes code for performing a transmission operation or a collection operation on part or all of the target data, wherein the third segment of code is used to increase the memory allowed to be used by the target computing platform.

24. The device according to claim 22, characterized in that The device is also used for: If the running parameter indicates that data skew occurs during the running of the second code set, determining whether the second code set includes code for performing a join operation on a table formed by part or all of the target data; In a case where the second code set includes the code for performing a join operation on a table formed by part or all of the data in the target data, searching the target data for a first data set and a second data set, wherein the magnitude of the data in the first data set is smaller than a preset first data magnitude and the data in the first data set is configured to perform the join operation, and the magnitude of the data in the second data set is greater than a preset second data magnitude and the data in the second data set is configured to perform the join operation, and the first data magnitude is smaller than the second data magnitude; When the first data set is found, broadcasting the first data set to multiple data processing nodes, so that the multiple data processing nodes perform the join operation on the table formed by the data in the first data set in parallel to obtain a first operation result; In the case where the second data set is found, splitting the data in the second data set into a first data subset having a data skew indicator and a second data subset not having a data skew indicator; Broadcasting the first data subset to the multiple data processing nodes, so that the multiple data processing nodes perform the join operation on the table formed by the data in the first data subset in parallel to obtain a second operation result; directly performing the join operation on the table formed by the data in the second data subset to obtain a third operation result; The second operation result and the third operation result are combined to obtain a fourth operation result.

25. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein the program can be executed by a terminal device or a computer to execute the method described in any one of claims 1 to 12.

26. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 12 through the computer program.

Images