Data processing method and related device

Abstract

The invention discloses a data processing method and a related device, and is applied to the technical field of computers. The method comprises the steps that the script running condition of a Lua environment running in a first thread is determined, a second thread is created when the script running condition meets a first condition, the second thread collects and stores memory data of the first thread at a first time point as first memory snapshot data, and the first thread obtains the first memory snapshot data from the second thread and stores the first memory snapshot data in the Lua environment. Second memory snapshot data is obtained, the second memory snapshot data is the memory data of the first thread at the second time point, and the first thread determines difference data according to the difference between the first memory snapshot data and the second memory snapshot data at different time points; by means of the mode, the newly-created thread is used for collecting and storing the memory snapshot data, the first thread does not need to collect and store the first memory snapshot data before difference calculation, consumption of a CPU and memory resources of the first thread is reduced, and the task processing efficiency of the first thread is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method and related apparatus. Background Technology

[0002] A thread is a term in operating system technology, representing the smallest unit of computation that an operating system can schedule. It is contained within a process and is the actual unit of operation within that process. A thread refers to a single, sequential flow of control within a process. A process can have multiple threads running in parallel, each executing different tasks concurrently. A single-threaded process has only one thread. In a single-threaded process, the program path is executed sequentially; earlier steps must be completed before later steps can be executed.

[0003] Current Lua performance analyzers run Lua environment scripts on the main thread and save a memory snapshot of the memory data within the main thread. When differential calculations are needed, they compare the memory snapshot data with the current memory data on the main thread to generate differential data, which helps identify performance bottlenecks and optimization opportunities in the Lua environment. However, collecting and saving memory snapshot data on the main thread consumes CPU and memory resources, which can block other tasks on the main thread and cause delays in their response. Summary of the Invention

[0004] This application provides a data processing method and related apparatus, in which a newly created thread collects and saves memory snapshot data. The first thread does not need to collect and save the first memory snapshot data before performing difference calculations, which reduces the CPU and memory resource consumption of the first thread and improves the task processing efficiency of the first thread.

[0005] In view of this, this application provides a data processing method, including:

[0006] Get the script execution status in the Lua environment within the first thread;

[0007] When the script execution meets the first preset condition, a second thread is created, which is different from the first thread.

[0008] The first memory snapshot data is collected and saved by the second thread. The first memory snapshot data is the memory data of the first thread at the first time point.

[0009] The first thread generates difference data based on the difference between the first memory snapshot data and the second memory snapshot data. The second memory snapshot data is the memory data of the first thread at a second time point, which is different from the first time point.

[0010] This application, in another aspect, provides a data processing apparatus, characterized in that it comprises:

[0011] The acquisition unit is used to acquire the script execution status in the Lua environment within the first thread;

[0012] A creation unit is used to create a second thread when the script execution meets the first preset condition. The second thread is different from the first thread.

[0013] The acquisition unit is used to acquire and save the first memory snapshot data through the second thread. The first memory snapshot data is the memory data of the first thread at the first time point.

[0014] The determining unit is used to generate difference data by a first thread based on the difference between the first memory snapshot data and the second memory snapshot data. The second memory snapshot data is the memory data of the first thread at a second time point, and the first time point is different from the second time point.

[0015] In one possible implementation, in another aspect of the embodiments of this application, the acquisition unit is further configured to:

[0016] The first thread collects the current memory data of the first thread as the second memory snapshot data.

[0017] In one possible implementation, in another aspect of the embodiments of this application, the creation unit is further configured to:

[0018] When the script execution meets the second preset condition, a third thread is created, which is different from the first and second threads.

[0019] The acquisition unit is also used for:

[0020] The second memory snapshot data is collected and saved through a third thread.

[0021] In one possible implementation, in another aspect of the embodiments of this application, the creation unit is specifically used for:

[0022] When the script execution meets the first preset condition, a first Lua state machine is created. The first Lua state machine is used to execute the second thread. The first thread is executed on the second Lua state machine. The first Lua state machine is different from the second Lua state machine.

[0023] In one possible implementation, in another aspect of the embodiments of this application, the data processing apparatus further includes a processing unit, which is specifically used for:

[0024] The second thread performs consistency processing on the first memory snapshot data to obtain the first processed data;

[0025] The third thread performs consistency processing on the second memory snapshot data to obtain the second processed data.

[0026] The specific unit is used for:

[0027] The first thread generates difference data based on the difference between the first processed data and the second processed data.

[0028] In one possible implementation, in another aspect of the embodiments of this application, the consistency processing includes one or more of the following: data cleaning, data standardization, data alignment, data transformation, data sorting, and error detection.

[0029] In one possible implementation, in another aspect of the embodiments of this application, the determination is specifically used for:

[0030] The first thread compares the first memory snapshot data with the second memory snapshot data to identify at least one data change corresponding to addition, deletion or modification.

[0031] Generate variance data that includes data changes.

[0032] Another aspect of this application provides a computer device, comprising:

[0033] Memory, transceiver, processor, and bus system;

[0034] The memory is used to store programs;

[0035] The processor is used to execute programs in memory, including methods for performing the aspects mentioned above;

[0036] Bus systems are used to connect memory and processor to enable communication between them.

[0037] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0038] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0039] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0040] This embodiment of the application determines the script execution status of the Lua environment running in the first thread. When the script execution status meets a first condition, a second thread is created. The second thread collects and saves the memory data of the first thread at a first time point as first memory snapshot data. The first thread can obtain this first memory snapshot data from the second thread, and then obtain second memory snapshot data, which is the memory data of the first thread at a second time point. Therefore, the first thread can determine the difference data based on the differences between the first and second memory snapshot data at different time points. Through this method, the newly created thread collects and saves the memory snapshot data, while the first thread does not need to collect and save the first memory snapshot data before performing difference calculations. This reduces the CPU and memory resource consumption of the first thread and improves its task processing efficiency. Attached Figure Description

[0041] Figure 1 This is an exemplary system architecture block diagram in an embodiment of this application;

[0042] Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of this application;

[0043] Figure 3 This is a schematic diagram of a process for determining difference data in an embodiment of this application;

[0044] Figure 4 This is a schematic diagram of an architecture for storing differential data in an embodiment of this application;

[0045] Figure 5 This is a schematic diagram of the structure of a data processing device according to an embodiment of this application;

[0046] Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0047] This application provides a data processing method and related apparatus, in which a newly created thread collects and saves memory snapshot data. The first thread does not need to collect and save the first memory snapshot data before performing difference calculations, which reduces the CPU and memory resource consumption of the first thread and improves the task processing efficiency of the first thread.

[0048] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0049] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0050] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0051] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0052] The following explains some terms used in the embodiments of this application.

[0053] LUA is a lightweight scripting language written in standard C and open-source. It is designed to be embedded in applications to provide flexible extension and customization capabilities.

[0054] Memory snapshot: refers to the state of data in memory at a specific moment. This is similar to a photograph; when you take a picture of a friend, a single photo can completely capture their image in a fleeting instant.

[0055] Current Lua performance analyzers run Lua environment scripts on the main thread and save a memory snapshot of the memory data within the main thread. When differential calculations are needed, they compare the memory snapshot data with the current memory data on the main thread to generate differential data, which helps identify performance bottlenecks and optimization opportunities in the Lua environment. However, collecting and saving memory snapshot data on the main thread consumes CPU and memory resources, which can block other tasks on the main thread and cause delays in their response.

[0056] To address the aforementioned issues, this application provides a data processing method. The method involves determining the script execution status of the Lua environment running in the first thread. When the script execution status meets a first condition, a second thread is created. The second thread collects and saves the memory data of the first thread at a first time point as first memory snapshot data. The first thread can obtain this first memory snapshot data from the second thread, and then obtain second memory snapshot data, which is the memory data of the first thread at a second time point. Therefore, the first thread can determine the difference data based on the differences between the first and second memory snapshot data at different time points. Through this method, the newly created thread collects and saves the memory snapshot data, eliminating the need for the first thread to collect and save the first memory snapshot data before performing difference calculations. This reduces the CPU and memory resource consumption of the first thread and improves its task processing efficiency.

[0057] This application applies to the field of artificial intelligence (AI). AI is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0058] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0059] The data processing method provided in this application can be implemented by various electronic devices. For example, it can be implemented by a terminal device alone, or by a server and a terminal device working together. For instance, a terminal device may execute the data processing method described below independently, or a terminal device and a server may work together to execute the data processing method described below. For example, a server may run a Lua environment in a first thread, a terminal device may obtain input from the user object, the server may update the running state of the Lua environment based on this input, and determine the running time of the Lua environment. When a first time point is reached, a second thread may be created to obtain the memory data of the first thread as first memory snapshot data. When a second time point is reached, a third thread may be created to obtain the memory data of the first thread as second memory snapshot data. The first thread may then determine the difference data based on the first and second memory snapshot data to identify performance bottlenecks and optimization opportunities in the Lua environment.

[0060] The electronic devices for data processing provided in this application can be various types of terminal devices or servers. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal devices can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these. The terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this.

[0061] Taking servers as an example, such as server clusters deployed in the cloud, AI as a Service (AIaaS) is provided to objects. The AIaaS platform will break down and execute several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI-themed marketplace. All objects can access and use one or more artificial intelligence services provided by the AIaaS platform through application programming interfaces.

[0062] The following description uses an example of a server and a terminal device collaboratively implementing the data processing method provided in the embodiments of this application. See also... Figure 1 , Figure 1 This is a schematic diagram of the architecture of an exemplary system 100 provided in an embodiment of this application. The terminal device 110 is connected to the server 130 through a network 120, which can be a wide area network, a local area network, or a combination of both.

[0063] In some embodiments, the terminal device 110 obtains input from the user based on the Lua environment and uploads it to the server 130. The server 130 updates the status of the Lua environment running on the first thread based on the input and determines the running time of the Lua environment. At a first time point, a second thread is created to obtain the memory data of the first thread as first memory snapshot data. At a second time point, the memory data of the first thread is obtained as second memory snapshot data. The first thread determines the difference data corresponding to the difference between the first memory snapshot data and the second memory snapshot data.

[0064] The data processing method provided in the embodiments of this application will be described below with reference to the accompanying drawings. The execution subject of the data processing method described below is a terminal device, which can be implemented by the terminal device by running the various computer programs mentioned above. Of course, based on the understanding of the following text, it is not difficult to see that the data processing method provided in the embodiments of this application can also be implemented by the terminal device and the server in collaboration.

[0065] Please see Figure 2 ,like Figure 2 The diagram shown is a flowchart illustrating a data processing method provided in an embodiment of this application. The method includes:

[0066] Step 201. Obtain the script execution status of the Lua environment in the first thread.

[0067] In this embodiment, the Lua environment refers to the specific context and configuration in which Lua code runs. It is designed to provide a stable and efficient execution platform for Lua scripts and can also be considered a "script execution framework." It includes the basic syntax rules of the Lua language, built-in function libraries, and interfaces for interacting with external systems. Within the Lua environment, code for game development, embedded systems, web development, automation scripts, scientific computing, and data analysis can be executed. Taking game development as an example, it can be used to implement game logic, AI, interface design, level design, control game characters, and expand game functionality. Terminal devices can use a Lua performance analyzer to analyze and optimize Lua code performance. After the script in this Lua environment runs as the first thread (the main thread), the script runtime in that Lua environment can be monitored in real time. In this embodiment, the executed time of the Lua environment can be directly obtained as the script runtime, or a timer can be used to determine whether the script runtime has reached the required time point; this is not limited here.

[0068] Step 202. When the script execution meets the first preset condition, create a second thread, which is different from the first thread.

[0069] In this embodiment, during the operation of the Lua environment, the state of the memory data of the first thread can be recorded as snapshot data. When the script execution meets the first preset condition, the terminal device can create a second thread and continue executing the Lua environment. The second thread and the first thread are threads that can be executed in parallel. The first preset condition can be timed, during data changes, during system backups, during user requests, during fault occurrences, or during performance testing, etc. For example, the timed method can be configured according to user needs. It can be generated at fixed time intervals, such as every hour, every day, every week, or every month, or it can be created at a preset specific time point, such as a fixed time every day (e.g., 5 AM), or a specific date and time every week. During data changes: When significant data changes occur, such as database updates or file modifications, a second thread can be created automatically. During system backups: A second thread can be created during system backups. During user requests: Users can manually trigger the creation of the second thread, especially before performing certain important operations, such as software upgrades or system migrations. During fault occurrences: When a system or data failure occurs, a second thread can be created immediately. When performing performance tests: A second thread can be created when conducting system performance tests or simulations.

[0070] Step 203. Collect and save the first memory snapshot data through the second thread. The first memory snapshot data is the memory data of the first thread at the first time point.

[0071] In this embodiment, after the second thread is created, the first thread returns to execute the script in the Lua environment. The second thread collects the memory data of the first thread at the first time point and saves it as snapshot data. This snapshot data can be called the first memory snapshot data. For example, taking game development as an example, snapshot data can be generated periodically during game map running. When the game malfunctions, the snapshot data can be used for quick data recovery, restoring the game to the state at the timed point in time.

[0072] Step 204. Generate difference data by the first thread based on the difference between the first memory snapshot data and the second memory snapshot data. The second memory snapshot data is the memory data of the first thread at the second time point, which is different from the first time point.

[0073] In this embodiment, the first thread can obtain the first memory snapshot data from the second thread, and then obtain the second memory snapshot data. The second memory snapshot data includes the memory data of the first thread at a second time point. The second time point can be before or after the first time point. The second memory snapshot data can be collected by the first thread or obtained by the first thread from other threads; this is not limited here. The terminal device can compare the differences between the first memory snapshot data and the second memory snapshot data using scripts in the Lua environment. For example, it can compare each data item in the first and second memory snapshot data one by one, identify the differences between the data items, classify the differences according to their nature, provide a detailed description for each difference item, including its location, value, change type, etc., and then integrate the difference items and their corresponding descriptions into difference data. Finally, it can use this difference data to identify the performance bottlenecks and optimization opportunities of the Lua environment.

[0074] This application embodiment determines the script execution status of the Lua environment running in the first thread. When the script execution status meets a first condition, a second thread is created. The second thread collects and saves the memory data of the first thread at a first time point as first memory snapshot data. The first thread can obtain this first memory snapshot data from the second thread, and then obtain second memory snapshot data. The second memory snapshot data is the memory data of the first thread at a second time point. Therefore, the first thread can determine the difference data based on the difference between the first and second memory snapshot data at different time points. Through this method, the newly created thread collects and saves the memory snapshot data, and the first thread does not need to collect and save the first memory snapshot data before performing difference calculations, reducing the CPU and memory resource consumption of the first thread and improving the task processing efficiency of the first thread.

[0075] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment provided by this application, before generating difference data by the first thread based on the difference between the first memory snapshot data and the second memory snapshot data, the method further includes:

[0076] The first thread collects the current memory data of the first thread as the second memory snapshot data.

[0077] In one or more embodiments, a method for obtaining second memory snapshot data is described. When performing difference calculations, the first thread can perform the calculation based solely on one snapshot data and the current memory data. That is, after obtaining the first memory snapshot data from the second thread, the first thread can collect its current memory data as the second memory snapshot data to perform difference calculations on the first and second memory snapshot data. Alternatively, the difference calculation can be performed by the second thread. The second thread collects and saves the first thread's memory data at a first time point as the first memory snapshot data. When difference calculations are needed, the second thread collects the first thread's memory data at a second time point as the second memory snapshot data and performs difference calculations on the first and second memory snapshot data.

[0078] This application's embodiments can be applied to loop structures that continuously run, process input, execute related logic, and generate output within a system, such as a game main loop, event loop, rendering loop, simulation loop, or server processing loop. The game main loop is responsible for continuously updating the game state and rendering the game screen. The event loop is a continuously running loop that waits for and processes user input events (such as mouse clicks and keyboard presses) and system events (such as timer events), and calls the corresponding handlers to respond to these events. The rendering loop is responsible for continuously updating and drawing the scene. The simulation loop is responsible for updating the simulation state based on the current simulation state and applied physical laws. In a web server, the server processing loop is responsible for receiving client requests, processing these requests (which may involve database queries, calculations, etc.), and sending responses back to the client.

[0079] For example, taking the main game loop as an example, the process for determining the difference data in this embodiment of the application can be referred to Figure 3As shown, step 301. Start the game engine on the Lua main thread. When the player starts the game, the game engine is also started. The game engine is responsible for various functions such as game operation, rendering, physics simulation, and audio processing. Step 302. Initialize the game configuration. The game engine reads the game configuration file, which usually contains various settings and parameters for the game. These settings may include graphics quality, sound effects settings, control methods, etc. The game engine parses these configurations and initializes the game's runtime environment according to the settings. Step 303. Initialize the Lua environment. Initializing the Lua environment usually involves embedding the Lua interpreter in the game engine and setting the corresponding environment variables and functions so that the game can use Lua scripts to extend its functionality or implement specific game logic. This process may include steps such as loading Lua libraries, creating a Lua state machine, and registering C functions to the Lua environment. Step 304. Enter the game's main loop. After initialization, the game enters the main loop, which will continue to run on the Lua main thread until the game is closed or exited. In each iteration of the game's main loop, the game engine checks the player's input, such as keyboard input, mouse movement, and clicks. This input is converted into commands or events that the game can understand. Step 305. Run the timer script logic in the game's main loop. When a timer triggers, it can call the corresponding function or execute the corresponding script code. This code can perform any necessary tasks, such as updating the game state, triggering events, or playing animations. After executing the script logic, it can also update the timer's state, such as the timer's next trigger time. Step 306. Determine if it's time to trigger the Lua snapshot tool. This is determined by the timer script. In each iteration of the game's main loop, the timer's state is checked. If the timer is already running and the current time exceeds its next trigger time, it's determined that it's time to trigger the Lua snapshot tool. If so, proceed to step 307; otherwise, proceed to step 304. Step 307. Execute the Lua snapshot tool by creating a Lua sub-thread. This Lua sub-thread can execute in parallel with the thread containing the game's main loop. Step 308. Collect the memory data of the current Lua main thread through the Lua sub-thread and save it as memory snapshot data. Step 309. When the Lua main thread needs to perform difference calculations, obtain the memory snapshot data of the Lua child thread and the memory data of the current main thread to perform difference calculations, generate difference data, and finally save the difference data to the local machine in the form of a text file, and then return to step 304.

[0080] This application provides a method for obtaining second memory snapshot data. Using this method, only one snapshot of data is used to calculate the difference between the snapshot and the current memory data, eliminating the need to create additional threads to consume resources.

[0081] Optionally, in the above Figure 2Based on the corresponding embodiments, in another optional embodiment provided by this application, before generating difference data by the first thread based on the difference between the first memory snapshot data and the second memory snapshot data, the method further includes:

[0082] When the script execution meets the second preset condition, a third thread is created, which is different from the first and second threads.

[0083] The second memory snapshot data is collected and saved through a third thread.

[0084] In one or more embodiments, a method for obtaining second memory snapshot data is described. The second preset condition is of the same type as the first preset condition, and can also be any one of the following: timed execution, data change, system backup, user request, fault occurrence, or performance testing. When the script execution meets the second preset condition, a third thread can be created. This third thread, along with the first and second threads, can be executed in parallel. At this time, the first thread returns to execute the script in the Lua environment, and the third thread collects the memory data of the first thread at a second time point and saves it as snapshot data. This snapshot data can be called the second memory snapshot data. For example, the first preset condition can be a fixed time in the timed execution. Assuming that the snapshot data is saved at 5 o'clock every day, that is, each time the terminal device creates a new thread to collect and save the memory data of the first thread at 5 o'clock as snapshot data when the script execution time in the Lua environment reaches 5 o'clock, the first preset condition can be that the script execution time in the Lua environment reaches 5 o'clock and no new thread is created. The second preset condition can be the same as the first preset condition.

[0085] For example, please refer to Figure 4 The diagram shown illustrates an architecture for storing differential data. Figure 4 When a player logs into the game, the terminal device can launch the game and enter the main game loop on the Lua main thread. When the player explores the map, loading new map resources or triggering new game events may cause data in memory to be updated or replaced. This embodiment of the application can use a timer script to periodically trigger the Lua snapshot tool. The Lua snapshot tool opens a Lua sub-thread to generate a snapshot of the data. Multiple triggers can generate multiple snapshots based on multiple Lua sub-threads, for example... Figure 4 The snapshot data generated and saved by sub-thread 1 (1), sub-thread 2 (2), sub-thread 3 (3), and sub-thread 4 (4) can be analyzed by the terminal device through the Lua main thread. Figure 4The difference analysis is performed on snapshot data 1 and snapshot data 2 to determine the difference data 1, and it is saved as difference data file 1. The difference analysis is performed on snapshot data 3 and snapshot data 4 to determine the difference data 2, and it is saved as difference data file 2.

[0086] This application provides a method for acquiring second memory snapshot data. By using this method, more new threads are created to collect and save snapshot data. The difference calculation of multiple snapshot data involves data snapshots at multiple time points, which can determine changes over a longer time span and enable more complex difference analysis and trend identification.

[0087] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment provided by this application, creating a second thread when the script execution meets the first preset condition includes:

[0088] When the script execution meets the first preset condition, a first Lua state machine is created. The first Lua state machine is used to execute the second thread. The first thread is executed on the second Lua state machine. The first Lua state machine is different from the second Lua state machine.

[0089] In one or more embodiments, a method for creating a second thread is described. Lua itself is a single-threaded scripting language and does not directly support multithreading. Instead, it uses coroutines to simulate concurrent execution. Coroutines are lightweight user-space threads that can be switched within Lua code but do not execute in parallel. Lua can be embedded into a multithreaded host language (such as C or Python), and threads can be created and managed within the host language, with each thread executing Lua code. Taking C as an example, a Lua state machine can be constructed, where each Lua state can run multiple coroutines. Each Lua state is independent, representing an independent Lua execution environment. From the C language perspective, a Lua virtual machine object is a Lua state. It represents the execution state of a Lua program and also refers to a Lua thread. The first thread can be executed on the second Lua state machine. When running the Lua environment, a first Lua state machine that is independent of the second Lua state machine can be created to execute the second thread. Since the first Lua state machine and the second Lua state machine are independent of each other, their execution actions do not affect each other, and the first thread and the second thread can be executed in parallel.

[0090] This application provides a method for creating a second thread. By creating a new Lua state machine to execute the new thread, the first thread and the new thread can execute in parallel, improving processing efficiency.

[0091] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment provided by this application, the method further includes:

[0092] The second thread performs consistency processing on the first memory snapshot data to obtain the first processed data;

[0093] The third thread performs consistency processing on the second memory snapshot data to obtain the second processed data.

[0094] The difference data generated by the first thread based on the difference between the first memory snapshot data and the second memory snapshot data includes:

[0095] The first thread generates difference data based on the difference between the first processed data and the second processed data.

[0096] In one or more embodiments, a method for generating difference data is described. Before determining the difference between first memory snapshot data and second memory snapshot data, each thread performs data processing, such as performing consistency processing on memory snapshot data collected and saved by different threads to unify the data format of different threads and facilitate data comparison between different threads. The second thread can process the first memory snapshot data according to a preset format, and the processed data can be called the first processed data. The third thread processes the second memory snapshot data according to the same preset format, and the processed data can be called the second processed data. Then, the first thread can directly obtain the first processed data from the second thread and the second processed data from the third thread, and only compare the difference between the first processed data and the second processed data to determine the difference data.

[0097] This application provides a method for generating difference data. Using this method, before the first thread compares the differences between snapshot data from different threads, the corresponding thread performs consistency processing on the snapshot data. This unifies the format of snapshot data collected by different threads, facilitates data comparison, accelerates the efficiency of the first thread's difference calculation, and eliminates the need for the first thread to perform consistency processing on the data, saving the first thread's resources and improving its task processing efficiency.

[0098] Optionally, in the above Figure 2Based on the corresponding embodiments, in another optional embodiment provided by the present application, the consistency processing includes one or more of the following: data cleaning, data standardization, data alignment, data transformation, data sorting, and error detection.

[0099] In one or more embodiments, a consistency processing method is described. Consistency processing may include one or more of the following: data cleaning, data standardization, data alignment, data transformation, data sorting, and error detection. Specifically, data cleaning may include one or more of the following: removing duplicate data, handling missing values, or filtering irrelevant data. Removing duplicate data may involve deduplicating duplicate records or redundant information that may be contained in the snapshot data. Handling missing values ​​may involve checking for missing or invalid values ​​in the data and performing fill, interpolation, or ignoring as appropriate. Filtering irrelevant data may involve retaining only data fields relevant to the difference calculation and removing fields unrelated to the calculation.

[0100] Data standardization can include one or more of the following: consistent data formatting or consistent data units. Consistent data formatting ensures that fields in snapshot data have consistent formats, such as date / time formats and numeric types. Consistent data units convert fields with different units in the data to have the same measurement standard.

[0101] Data alignment can include one or more of timestamp alignment and data dimension alignment. Timestamp alignment ensures that timestamps between different snapshots are aligned, enabling correct comparisons of time series. Data dimension alignment ensures that the correspondences between the multiple dimensions (such as spatial location, resource distribution, etc.) contained in the snapshot data are correct.

[0102] Data transformation can include one or more of the following: encoding transformation or data aggregation. Encoding transformation can be the conversion of text data into a computationally suitable encoding format, such as converting categorical data into numeric codes. Data aggregation can be the aggregation of data as needed, such as calculating averages, sums, etc., to simplify the difference calculation process.

[0103] Data sorting can sort data based on time or other key fields to ensure that differences are compared in the correct order during calculation.

[0104] Error detection can verify the integrity and consistency of data by checking for outliers or erroneous data and taking appropriate action.

[0105] This application provides a consistency processing method. By using one or more of the following methods—data cleaning, data standardization, data alignment, data transformation, data sorting, and error detection—to process snapshot data, it is possible to ensure that discrepancy calculations are based on high-quality, reliable datasets. This not only improves the accuracy of discrepancy calculations but also reduces errors and interference during the calculation process. Furthermore, optimizing the data processing flow can improve computational efficiency, providing more timely and accurate information support for decision-making and analysis.

[0106] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment provided by this application, generating difference data by a first thread based on the difference between the first memory snapshot data and the second memory snapshot data includes:

[0107] The first thread compares the first memory snapshot data with the second memory snapshot data to identify at least one data change corresponding to addition, deletion or modification.

[0108] Generate variance data that includes data changes.

[0109] In one or more embodiments, a method for obtaining difference data is described. Appropriate comparison algorithms or tools can be used to compare two memory snapshots of data. The comparison process can be an item-by-item comparison or a hash- or signature-based comparison to identify all differences between the two memory snapshots of data and categorize the identified differences. For example, a new item: a data item that exists in the second memory snapshot but not in the first memory snapshot. A deleted item: a data item that exists in the first memory snapshot but not in the second memory snapshot. A modified item: a data item that exists in both snapshots but whose value or state has changed.

[0110] For example, taking the first memory snapshot data as the earliest point in time, the memory snapshot data can be stored in the form of a table. The data in the table is saved in the form of key-value pairs. The first memory snapshot data can correspond to the first table, and the second memory snapshot data can correspond to the second table. Accordingly, the keys in the first table and the second table can be traversed. Only the key in the first table corresponds to the data that has been deleted, only the key in the second table corresponds to the data that has been added, and the key that exists in both tables but has different values ​​corresponds to the data that has been modified.

[0111] This application provides a method for obtaining difference data. This method accurately identifies data changes (additions, deletions, or modifications) between different memory snapshots, improving the intuitiveness and accuracy of the difference data.

[0112] The data processing method has been described above; the apparatus for executing this method will be described below.

[0113] like Figure 5 The diagram shown is a structural schematic of a data processing apparatus provided in an embodiment of this application. The apparatus 50 includes:

[0114] The acquisition unit 501 is used to acquire the script execution status of the Lua environment in the first thread;

[0115] Creation unit 502 is used to create a second thread when the script execution meets the first preset condition. The second thread is different from the first thread.

[0116] The acquisition unit 503 is used to acquire and save the first memory snapshot data through the second thread. The first memory snapshot data is the memory data of the first thread at the first time point.

[0117] The determination unit 504 is used to generate difference data by the first thread based on the difference between the first memory snapshot data and the second memory snapshot data. The second memory snapshot data is the memory data of the first thread at a second time point, and the first time point is different from the second time point.

[0118] In this embodiment of the application, a data processing apparatus is provided. With this apparatus, a newly created thread collects and saves memory snapshot data. The first thread does not need to collect and save first memory snapshot data before performing difference calculations, thus reducing the CPU and memory resource consumption of the first thread and improving its task processing efficiency.

[0119] Optionally, in the above Figure 5 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 50 provided in this application, the acquisition unit 503 is further used for:

[0120] The first thread collects the current memory data of the first thread as the second memory snapshot data.

[0121] In this application embodiment, a data processing apparatus is provided. With this apparatus, difference calculations are performed only based on a snapshot of data and the current in-memory data, eliminating the need to create additional threads to consume resources.

[0122] Optionally, in the above Figure 5 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 50 provided in this application, the creation unit 502 is further configured to:

[0123] When the script execution meets the second preset condition, a third thread is created, which is different from the first and second threads.

[0124] The acquisition unit 503 is also used for:

[0125] The second memory snapshot data is collected and saved through a third thread.

[0126] In this application embodiment, a data processing apparatus is provided. Through this apparatus, more new threads are created to collect and save snapshot data. The difference calculation of multiple snapshot data involves data snapshots at multiple time points, which can determine changes over a longer time span and perform more complex difference analysis and trend identification.

[0127] Optionally, in the above Figure 5 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 50 provided in this application, the creation unit 502 is specifically used for:

[0128] When the script execution meets the first preset condition, a first Lua state machine is created. The first Lua state machine is used to execute the second thread. The first thread is executed on the second Lua state machine. The first Lua state machine is different from the second Lua state machine.

[0129] In this embodiment of the application, a data processing apparatus is provided. Using this apparatus, a new Lua state machine is created to execute a new thread, allowing the first thread and the new thread to execute in parallel, thereby improving processing efficiency.

[0130] Optionally, in the above Figure 5 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 50 provided in this application, the data processing apparatus 50 further includes a processing unit 505, which is specifically used for:

[0131] The second thread performs consistency processing on the first memory snapshot data to obtain the first processed data;

[0132] The third thread performs consistency processing on the second memory snapshot data to obtain the second processed data.

[0133] Unit 504 is specifically used for:

[0134] The first thread generates difference data based on the difference between the first processed data and the second processed data.

[0135] In this application embodiment, a data processing apparatus is provided. With this apparatus, before the first thread compares the differences between snapshot data from different threads, the corresponding threads perform consistency processing on the snapshot data. This unifies the format of snapshot data collected by different threads, facilitates data comparison, accelerates the efficiency of the first thread's difference calculation, and eliminates the need for the first thread to perform consistency processing on the data, saving the first thread's resources and improving its task processing efficiency.

[0136] Optionally, in the above Figure 5 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 50 provided in this application, consistency processing includes one or more of data cleaning, data standardization, data alignment, data transformation, data sorting, and error detection.

[0137] This application provides a data processing apparatus. By using this apparatus to process snapshot data using one or more of the following methods: data cleaning, data standardization, data alignment, data transformation, data sorting, and error detection, it is possible to ensure that difference calculations are performed on high-quality, reliable datasets. This not only improves the accuracy of difference calculations but also reduces errors and interference during the calculation process. Furthermore, optimizing the data processing flow can improve computational efficiency, providing more timely and accurate information support for decision-making and analysis.

[0138] Optionally, in the above Figure 5 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 50 provided in this application, the specific 504 is used for:

[0139] The first thread compares the first memory snapshot data with the second memory snapshot data to identify at least one data change corresponding to addition, deletion or modification.

[0140] Generate variance data that includes data changes.

[0141] This application provides a data processing apparatus. This apparatus accurately identifies data changes (additions, deletions, or modifications) between different memory snapshots, improving the intuitiveness and accuracy of the difference data.

[0142] Figure 6 This is a schematic diagram of a computer device structure provided in an embodiment of this application. The computer device 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the computer device 300.

[0143] Computer device 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0144] The steps performed by the terminal device in the above embodiments can be based on this Figure 6 The computer device structure shown.

[0145] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0146] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Get the script execution status in the Lua environment within the first thread; When the script execution meets the first preset condition, a second thread is created, which is different from the first thread. The second thread collects and saves the first memory snapshot data, which is the memory data of the first thread at the first time point. The first thread generates difference data based on the difference between the first memory snapshot data and the second memory snapshot data. The second memory snapshot data is the memory data of the first thread at a second time point, and the first time point is different from the second time point.

2. The method according to claim 1, characterized in that, Before generating difference data based on the difference between the first memory snapshot data and the second memory snapshot data via the first thread, the method further includes: The first thread collects its current memory data as the second memory snapshot data.

3. The method according to claim 1, characterized in that, Before generating difference data based on the difference between the first memory snapshot data and the second memory snapshot data via the first thread, the method further includes: When the script execution meets the second preset condition, a third thread is created, which is different from the first thread and the second thread. The second memory snapshot data is collected and saved through the third thread.

4. The method according to claim 1, characterized in that, The step of creating a second thread when the script execution meets the first preset condition includes: When the script execution meets the first preset condition, a first Lua state machine is created. The first Lua state machine is used to execute the second thread. The first thread is executed on the second Lua state machine. The first Lua state machine is different from the second Lua state machine.

5. The method according to claim 3, characterized in that, The method further includes: The second thread performs consistency processing on the first memory snapshot data to obtain the first processed data; The third thread performs consistency processing on the second memory snapshot data to obtain the second processed data. The step of generating difference data by the first thread based on the difference between the first memory snapshot data and the second memory snapshot data includes: The difference data is generated by the first thread based on the difference between the first processed data and the second processed data.

6. The method according to claim 5, characterized in that, The consistency processing includes one or more of the following: data cleaning, data standardization, data alignment, data transformation, data sorting, and error detection.

7. The method according to claim 1, characterized in that, The step of generating difference data by the first thread based on the difference between the first memory snapshot data and the second memory snapshot data includes: The first thread compares the first memory snapshot data and the second memory snapshot data to identify at least one data change corresponding to addition, deletion, or modification. Generate the difference data that includes the changes in the data.

8. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire the script execution status in the Lua environment within the first thread; A creation unit is used to create a second thread when the script execution meets a first preset condition, wherein the second thread is different from the first thread; The acquisition unit is used to acquire and save first memory snapshot data through the second thread, wherein the first memory snapshot data is the memory data of the first thread at a first time point; The determining unit is used to generate difference data by the first thread based on the difference between the first memory snapshot data and the second memory snapshot data, wherein the second memory snapshot data is the memory data of the first thread at a second time point, and the first time point is different from the second time point.

9. A computer device, characterized in that, include: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, including performing the method as described in any one of claims 1 to 7; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

10. A computer-readable storage medium comprising instructions, when executed on a computer, causing the computer to perform the method as claimed in any one of claims 1 to 7.

11. A computer program product, characterized in that, When the computer program product is executed on a computer, the computer performs the method as described in any one of claims 1 to 7.

Images