Data processing method and device, equipment, storage medium and program product

Abstract

The application provides a data processing method, device, equipment, storage medium and program product, comprising: a first processing process receives a data processing request for a target object, the data processing request is used to request target type data processing on target data corresponding to the target object; in response to the data processing request, index information corresponding to the target data is determined; the index information is sent to a second processing process, the second processing process is used to store shared data of at least two first processing processes in a tree-shaped storage structure; based on the index information, the target data is indexed in the tree-shaped storage structure through the second processing process, and the target data is returned to the first processing process; the target type data processing is performed on the target data through the first processing process. Through the application, the defects of the shared data storage structure can be improved, and the performance of data storage is improved.

Description

Technical Field

[0001] This application relates to Internet technology, and more particularly to a data processing method, apparatus, device, computer-readable storage medium, and computer program product. Background Technology

[0002] In related technologies, there are two main solutions for inter-process data sharing in Swoole: the first is to use Swoole's natively supported swoole_table; the second is to use external storage, such as a database or cache server. However, for the first solution, swoole_table stores data in key / value format and cannot store complex data structures such as multidimensional arrays, thus limiting its use cases. Furthermore, the memory usage of swoole_table is pre-calculated before data storage and cannot be dynamically adjusted during storage, leading to uneven memory allocation and impacting storage performance. For the second solution, using external storage requires a cross-machine network call, which is generally inefficient and affects Swoole performance, making it unsuitable for performance-critical scenarios. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the defects of shared data storage structure and enhance data storage performance.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data processing method, including:

[0006] The first processing process receives a data processing request for a target object, the data processing request being used to request target data of the target object to be processed according to the target type;

[0007] In response to the data processing request, determine the index information corresponding to the target data;

[0008] The index information is sent to the second processing process, which is used to store shared data of at least two first processing processes in a tree storage structure with the application as the root node.

[0009] Based on the index information, the second processing process indexes the target data in the tree storage structure and returns the target data to the first processing process;

[0010] The target data is processed using the first processing step, which performs the target type of data processing on the target data.

[0011] This application provides a data processing apparatus, including:

[0012] The first processing module is configured to receive a data processing request for a target object, wherein the data processing request is used to request target data of the target object to be processed in a target type.

[0013] The response module is used to determine the index information corresponding to the target data in response to the data processing request;

[0014] A sending module is used to send the index information to a second processing process, wherein the second processing process is used to store shared data of at least two first processing processes in a tree storage structure;

[0015] The second processing module is used to obtain the target data by indexing the tree storage structure through the second processing process based on the index information, and return the target data to the first processing process;

[0016] An execution module is used to perform data processing of the target type on the target data through the first processing process.

[0017] In the above scheme, the response module is further configured to respond to the data processing request, parse the data processing request, obtain the object identifier of the target object, and the node identifier of the parent index node to which the object identifier belongs when the object identifier is the index node; and determine the object identifier and the node identifier as the index information corresponding to the target data.

[0018] In the above scheme, the second processing module is further configured to, when the data processing request is sent by an instant messaging application, the target object is a session object, and the parent index node is a session room, determine a tree storage structure with the instant messaging application as the root node through the second processing process; in the tree storage structure with the instant messaging application as the root node, index the session room to which the target object belongs based on the room identifier of the session room; index the session connection descriptor corresponding to the target object based on the session room and the object identifier of the session object, and use the session connection descriptor as the target data.

[0019] In the above scheme, the second processing module is further configured to, when the data processing request is sent by a media information application, the target object is the associated information of the media information, and the parent index node is the media information, determine a tree storage structure with the media information application as the root node through the second processing process; in the tree storage structure with the media information application as the root node, index the media information to which the target object belongs based on the media identifier of the media information; index the information content of the associated information based on the information identifier of the media information and the associated information, and use the information content of the associated information as the target data.

[0020] In the above scheme, the index information includes at least two levels. The second processing module is further used to perform layer-by-layer indexing on the tree storage structure based on the parent index information in the at least two levels of index information to determine the tree node corresponding to the parent index information; and to index the target data corresponding to the target object based on the tree node corresponding to the parent index information and the sub-index information in the at least two levels of index information.

[0021] In the above scheme, the execution module is further configured to obtain the target type for data processing of the target data; wherein the target type includes at least one of data addition, data deletion, data query, and data modification; and to perform data processing on the target data based on the target type.

[0022] In the above scheme, the device further includes a first storage module, which is used to: acquire the target data to be stored and the number of the second processing processes; when the number of the second processing processes is at least two, determine the second processing process for storing the target data; and store the target data in the tree storage structure of the determined second processing process.

[0023] In the above scheme, the first storage module is further configured to perform hash processing on the target data to obtain a processing result when the number of the second processing processes is at least two; obtain the identifier of each second processing process; match the processing result with the identifier to obtain a matching result; and select a second processing process for storing the target data from at least two second processing processes based on the matching result.

[0024] In the above scheme, the device further includes a second storage module, which is used to acquire the target data to be stored and the number of the second processing processes; when the number of the second processing processes is one, randomly generate an identifier corresponding to the second processing process; and store the target data in the tree storage structure of the second processing process based on the identifier.

[0025] In the above scheme, the number of the second processing processes is at least two. The sending module is further configured to obtain the mapping relationship between the index information and each second processing process; determine the identifier of the second processing process corresponding to the index information based on the mapping relationship; and send the index information to the corresponding second processing process based on the identifier of the second processing process.

[0026] This application provides an electronic device, including:

[0027] Memory, used to store executable instructions;

[0028] The processor, when executing executable instructions stored in the memory, implements the data processing method provided in the embodiments of this application.

[0029] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the data processing method provided in this application.

[0030] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the data processing method provided in this application.

[0031] The embodiments of this application have the following beneficial effects:

[0032] By setting up a first processing process and a second processing process, when the first processing process receives a data processing request, it determines the index information corresponding to the request. Then, the second processing process determines the target data corresponding to the request based on the index information. Finally, the first processing process performs target-type data processing on the target data. In this way, the first processing process handles the data processing request, while the second processing process stores and manages the shared data among the first processing processes. The combination of the two processing processes achieves data sharing and corresponding shared data processing, improving the shortcomings of shared data storage structures and enhancing data storage performance. Attached Figure Description

[0033] Figure 1 This is an optional architecture diagram of the data processing system 100 provided in this application embodiment;

[0034] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0035] Figure 3This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of the tree storage structure of the second processing process provided in the embodiments of this application;

[0037] Figure 5 This is a schematic diagram of the tree storage structure of the second processing process provided in the embodiments of this application;

[0038] Figure 6 This is a schematic diagram illustrating the processing of target data storage provided in an embodiment of this application;

[0039] Figure 7 This is a schematic diagram illustrating the processing of target data storage provided in an embodiment of this application;

[0040] Figure 8 This is a flowchart illustrating the synchronous processing of the data processing method provided in the embodiments of this application;

[0041] Figure 9 This is an asynchronous processing flowchart of the data processing method provided in the embodiments of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first," "second," and "third" are used merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first," "second," and "third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. In the following description, the term "multiple" means at least two.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0046] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0047] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0048] 1) Swoole: A PHP extension, an asynchronous, parallel, high-performance network communication engine for PHP, providing an asynchronous multi-threaded server for the PHP language, i.e., a PHP asynchronous coroutine framework.

[0049] 2) swoole_table, a high-performance, concurrent data structure based on shared memory and locks, is used to solve data sharing and synchronization locking problems in multi-process or multi-threaded environments, i.e., a high-performance, high-concurrency structure. Using swoole_table, data can be shared easily without worrying about data synchronization issues, without requiring user-level locking, and without considering the overhead of locks.

[0050] 3) WebSocket is an application layer protocol specifically designed to provide full-duplex communication between web applications and servers.

[0051] 4) h5, the fifth major revision of HTML, also refers to all digital products made using the HTML5 language.

[0052] The applicant discovered that traditional inter-process data sharing in Swoole primarily employs two methods: The first is using Swoole's natively supported swoole_table. Specifically, swoole_table is typically pre-initialized before the Swoole server starts, and then the server itself is started. After startup, each data processing process can directly manipulate the previously generated swoole_table to perform CRUD operations, achieving inter-process data sharing. The second method uses external storage, such as a database or cache server. Each data processing process accesses the external storage server to store and share data.

[0053] However, regarding the first approach, `swoole_table` stores data in a key / value format, which cannot store complex data structures such as multidimensional arrays, thus limiting its use cases. Furthermore, `swoole_table` needs to be initialized and its space allocated in the master process before the Swoole service starts. The amount of storage space needs to be pre-calculated during allocation, and the table size cannot be modified during use. This means that if too much space is allocated, it will result in wasted memory; if too little space is allocated, the application may not have enough space to use it. Regarding the second approach, using external storage requires a cross-machine network call, which is generally inefficient and will impact Swoole performance, making it unsuitable for performance-critical scenarios.

[0054] Based on this, embodiments of this application provide a data processing method, apparatus, device, computer-readable storage medium, and computer program product. Based on the process mode supported by native Swoole, a task management process is started to store and process multi-dimensional complex data that needs to be shared between various data processing processes. This makes up for the shortcomings of Swoole_table in being unable to store complex multi-dimensional data structures and the lack of memory scalability. Moreover, it has a significant performance improvement over the external storage solution, which can improve the defects of shared data storage structure and improve data storage performance.

[0055] See Figure 1 , Figure 1 This is an optional architecture diagram of the data processing system 100 provided in this application embodiment. The terminal (terminal 400 is shown as an example) connects to the server 200 through the network 300, which can be a wide area network, a local area network, or a combination of both. As an example, in a data processing application scenario, a websocket server is started using Swoole. The user establishes a long connection with the server on the H5 page via the websocket protocol. After the long connection is established, the server maintains user information and the connection descriptor of the long connection. When a chat application needs to push messages to all users in a chat group, the first processing process receives the data push request for the user, determines the user identifier of each user, and then, based on the user identifier, the second processing process indexes the user information and the connection descriptor of the long connection in the tree storage structure, and sends the connection descriptor to the first processing process. The first processing process then executes the message push for the target user based on the connection descriptor.

[0056] Terminal 400 is used by users to access client 401 and is displayed on display interface 401-1 (display interface 401-1 is shown as an example). Terminal 400 and server 200 are interconnected via wired or wireless network.

[0057] Server 200 is used to receive a data processing request for a target object through a first processing process. The data processing request requests data processing of the target type for the target data corresponding to the target object. In response to the data processing request, it determines the index information corresponding to the target data. It sends the index information to a second processing process, which uses a tree storage structure with the application as the root node to store the shared data of at least two first processing processes. Based on the index information, the second processing process indexes the target data in the tree storage structure and returns the target data to the first processing process. The first processing process performs data processing of the target type on the target data, obtains the processing result, and returns the processing result to terminal 400.

[0058] Terminal 400 is also used to display the received processing results on display interface 401-1.

[0059] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, set-top box, or mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, in-vehicle terminal, in-vehicle infotainment system, in-vehicle data center, portable gaming device, smart speaker, and smartwatch), but is not limited thereto. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0060] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. In practical applications, the electronic device can be... Figure 1 The server 200 or terminal 400 shown are described in the following document. Figure 2 , Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2The general labeled all buses as Bus System 440.

[0061] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0064] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0065] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0066] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0067] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0068] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.).

[0069] The input processing module 454 is used to detect and translate one or more user inputs or interactions from any one of the input devices 432.

[0070] In some embodiments, the data processing apparatus provided in this application can be implemented in software. Figure 2 A data processing device 455 stored in memory 450 is shown. It may be software in the form of programs and plug-ins, including the following software modules: a first processing module 4551, a response module 4552, a sending module 4553, a second processing module 4554, and an execution module 4555. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement.

[0071] In other embodiments, the data processing apparatus provided in this application can be implemented in hardware. As an example, the data processing apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0072] In some embodiments, the terminal or server can implement the data processing method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP or a web browser APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0073] Based on the foregoing description of the data processing system and electronic device provided in the embodiments of this application, the data processing method provided in the embodiments of this application will be described below. In actual implementation, the data processing method provided in the embodiments of this application can be implemented by a terminal or a server alone, or by a terminal and a server working together, so that... Figure 1 The following description uses the example of server 200 executing the data processing method provided in this embodiment of the application independently. See also... Figure 3 , Figure 3 This is a flowchart illustrating the data processing method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0074] Step 101: The first processing process in the server receives a data processing request for the target object. The data processing request is used to request data processing of the target type for the target data corresponding to the target object.

[0075] In practice, the server first detects data processing requests for a target object triggered by the user. When the server detects a data processing request for a target object, it sends the detected data processing request to the first processing process, so that the first processing process receives the data processing request for the target object.

[0076] It should be noted that the data processing request here is used to request data processing of the target type on the target data corresponding to the target object, and the target type here can include at least one of data addition, data deletion, data query and data modification.

[0077] Step 102: In response to the data processing request, determine the index information corresponding to the target data.

[0078] In practice, after receiving a data processing request, the first processing process parses the request to obtain the index information corresponding to the target data. Specifically, in response to the data processing request, the process parses the request to obtain the object identifier of the target object and, when the object identifier is the index node, the node identifier of the parent index node to which the object identifier belongs. Then, the object identifier and the node identifier are determined as the index information corresponding to the target data.

[0079] Step 103: Send the index information corresponding to the target data to the second processing process. The second processing process is used to store the shared data of at least two first processing processes using a tree storage structure.

[0080] In practice, the method by which the first processing process sends index information to the second processing process differs depending on the number of second processing processes. The following section describes two implementation processes for sending the index information corresponding to the target data to the second processing process, taking into account the difference in the number of second processing processes.

[0081] In some embodiments, when there is only one second processing process, the identifier of the second processing process is obtained, and the index information is sent to the second processing process based on the identifier. Here, the identifier is a randomly generated identifier when storing the target data.

[0082] In some embodiments, when the number of second processing processes is at least two, the identifier of the second processing process corresponding to the index information is determined based on the index information, and then the index information is sent to the second processing process corresponding to the identifier based on the identifier. Specifically, the process of determining the identifier of the second processing process corresponding to the index information based on the mapping relationship involves obtaining the identifier of each second processing process, and then determining the second processing process corresponding to the index information and its corresponding identifier based on the mapping relationship. It should be noted that the mapping relationship between the index information and each second processing process is obtained during the process of storing the target data in the tree storage structure of the corresponding second processing process.

[0083] The process of storing the target data will be explained in detail in step 104.

[0084] It should be noted that the first processing process can first perform homomorphic encryption on the parsed index information, and then send the encrypted index information to the second processing process, thereby improving the security of data transmission.

[0085] Step 104: Based on the index information, the target data is obtained by indexing in the tree storage structure through the second processing process, and the target data is returned to the first processing process.

[0086] In practice, after obtaining the index information, the second processing process determines the number of levels in the corresponding tree storage structure included in the index information. When the number of levels in the index information is at least two, the tree storage structure is indexed layer by layer based on the parent index information in the at least two levels of index information to determine the tree node corresponding to the parent index information; based on the tree node corresponding to the parent index information and the sub-index information in the at least two levels of index information, the target data corresponding to the target object is obtained.

[0087] As an example, see Figure 4 , Figure 4This is a schematic diagram of the tree storage structure of the second processing process provided in this application embodiment. Application 1 is an instant messaging application, the target object is a session object, the parent index node is the session room and application, and fd is the stored target data, i.e., the session connection descriptor corresponding to the target object. Specifically, when the data processing request is sent by the instant messaging application, the target object is a session object, and the parent index node is a session room, the second processing process determines that the corresponding application is an instant messaging application based on the application identifier included in the index information. Once the corresponding instant messaging application is determined, a tree storage structure with the instant messaging application as the root node can be determined. In the tree storage structure corresponding to the instant messaging application, the session room to which the target object belongs is indexed based on the room identifier of the session room; the session connection descriptor corresponding to the target object is indexed based on the session room and the object identifier of the session object, and the session connection descriptor is used as the target data.

[0088] As another example, see Figure 5 , Figure 5 This is a schematic diagram of the tree storage structure of the second processing process provided in this application embodiment. Application n is the media information application, the target object is the associated information corresponding to the media information, the parent index node is the media information and the application, and the content information is the stored target data, i.e., the information content corresponding to the target object. Specifically, when a data processing request is sent by the media information application, the target object is the associated information of the media information, and the parent index node is the media information, the second processing process determines the corresponding application as the media information application based on the application identifier included in the index information. Once the corresponding media information application is determined, a tree storage structure with the media information application as the root node can be determined. In the tree storage structure with the media information application as the root node, the media information to which the target object belongs is indexed based on the media identifier of the media information; the information content of the associated information is indexed based on the information identifiers of the media information and the associated information, and the information content of the associated information is used as the target data.

[0089] In some embodiments, before obtaining the target data based on the index information, the target data needs to be stored in the tree storage structure of the second processing process. Specifically, firstly, the target data to be stored and the number of second processing processes are obtained, and the number of second processing processes is determined. Based on the determination result, the target data is stored in the tree storage structure of the corresponding second processing process. Here, two target data storage methods are used to address the difference in the number of second processing processes. The two target data storage methods will be described below.

[0090] In some embodiments, when the number of second processing processes is one, an identifier corresponding to the second processing process is randomly generated; based on this identifier, the target data is stored in the tree storage structure of the second processing process, see [link to documentation]. Figure 6 , Figure 6 This is a schematic diagram of the target data storage process provided in this application embodiment. When there is only one second processing process, an identifier corresponding to the second processing process is randomly generated, and the target data is stored in the tree storage structure of the second processing process. Thus, after the first processing process analyzes and obtains the index information, it directly sends the index information to the second processing process based on the randomly generated identifier, so that the second processing process can index the target data corresponding to the target object.

[0091] In some embodiments, when the number of second processing processes is at least two, a second processing process for storing the target data is determined; the target data is stored in the tree storage structure of the determined second processing process. Specifically, see... Figure 7 , Figure 7 This is a schematic diagram illustrating the processing of target data storage provided in an embodiment of this application, based on... Figure 7 The process of determining the second processing process for storing target data can be as follows: hashing the target data to obtain a processing result; obtaining the identifier of each second processing process; matching the processing result with the identifier to obtain a matching result; and selecting the second processing process for storing the target data from at least two second processing processes based on the matching result. Specifically, firstly, a process identifier corresponding to each second processing process is pre-defined, where the range of the process identifier is [0, n-1], and n is the number of second processing processes. Then, the target data is hashed to obtain a processing result. Next, the last character of the processing result corresponding to each target data is selected as the matching result representation character corresponding to each target data. The matching result representation character corresponding to each target data is then matched with the identifier of each second processing process. Based on the matching result, the second processing process for storing each target data is selected from at least two second processing processes.

[0092] It should be noted that when a matching result character corresponds to the identifier of a second processing process, the corresponding second processing process is determined as the second processing process for storing the target data corresponding to that matching result character. When a matching result character does not correspond to the identifier of any second processing process, a second processing process is randomly selected as the second processing process for storing the target data corresponding to that matching result character. Here, the random selection method can be to select the second processing process with the smallest identifier, the largest identifier, an odd-numbered identifier, or an even-numbered identifier. There are no restrictions on the method of randomly selecting the second processing process.

[0093] As an example, this explanation assumes there are three secondary processing processes. Let the process identifiers of the three secondary processing processes be {0, 1, 2}, and the target data consist of five groups. After hashing the five groups of target data, the last character is taken, resulting in matching result representation characters {1, 2, 3, 4, 5} for each group of target data. Each group's matching result representation character is then matched against the identifier of the secondary processing process. The results show that matching result representation character 1 matches identifier 1, matching result representation character 2 matches identifier 2, while characters {3, 4, 5} do not match any of the identifiers. Therefore, the target data corresponding to the matching result character 1 is stored in the tree storage structure of the second processing process corresponding to the identifier 1, and the target data corresponding to the matching result character 2 is stored in the tree storage structure of the second processing process corresponding to the identifier 2. Then, the target data corresponding to the matching result characters {3, 4, 5} are stored in the tree storage structure of the randomly selected second processing process. For example, the second processing process with the smallest identifier, i.e., the second processing process with the identifier 0, is selected to store the target data corresponding to the matching result characters {3, 4, 5} in the tree storage structure of the second processing process corresponding to the identifier 0.

[0094] It should be noted that when there are at least two second processing processes, the process of determining the second processing process for storing the target data can be as follows: Besides hashing the target data and determining the second processing process based on the processing result, it can also be based on the load of the tree-structured storage of the second processing processes. For example, the second processing process with the lowest load among multiple second processing processes can be selected as the second processing process for storing the target data. Alternatively, the target data can be initially stored in a fixed location among multiple second processing processes. When that second processing process is fully loaded, another second processing process is selected as the second processing process for storing the target data, and this process continues until the selected second processing process is fully loaded, at which point the next second processing process is selected as the second processing process for storing the target data.

[0095] Alternatively, the corresponding second processing process can be selected based on the order in which the target data is received. For example, for n second processing processes, when the first target data is received, the target data is stored in the tree storage structure of the first second processing process; when the second target data is received, the target data is stored in the tree storage structure of the second second processing process, and so on, until the nth target data is received, at which point the target data is stored in the tree storage structure of the nth second processing process. Then, when the (n+1)th target data is received, the target data is stored in the tree storage structure of the first of the multiple second processing processes, and this process is repeated. This application does not limit the method of determining the second processing process for storing the target data when the number of second processing processes is at least two.

[0096] In this way, by storing the target data through the tree storage structure of the second processing process, the deficiency of swoole_table in being unable to store complex multidimensional data structures is compensated. At the same time, by storing different target data in the tree storage structures of different second processing processes, the accurate retrieval of target data can be ensured when reading data.

[0097] In actual implementation, the data processing method provided in this application embodiment can be implemented by a synchronous processing model or an asynchronous processing model.

[0098] In some embodiments, see Figure 8 , Figure 8 This is a flowchart illustrating the synchronization process of the data processing method provided in this application embodiment, based on... Figure 8 The user triggers a data processing request, which is then sent by the server to the first processing process. Upon receiving the request, the first processing process analyzes it to obtain the index information corresponding to the target data and sends this index information to the second processing process. The first process then waits for the second processing process to send the target data. Only after receiving the target data from the second processing process does the first processing process continue with subsequent processing. This synchronous processing model avoids deadlocks and dirty data reads, reducing the possibility of errors during data processing.

[0099] In some embodiments, see Figure 9 , Figure 9This is an asynchronous processing flowchart of the data processing method provided in this application embodiment. A user triggers a data processing request, which is then sent by the server to a first processing process. Upon receiving the request, the first processing process analyzes it to obtain the index information corresponding to the target data and sends this index information to a second processing process. Then, without waiting for the second processing process to send the target data, the process continues with subsequent processing. Thus, by implementing the data processing process through an asynchronous model, the efficiency of data processing is improved.

[0100] Step 105: Perform target type data processing on the target data through the first processing process.

[0101] In practice, when the first processing process analyzes the received data processing request for the target object, it can determine the target type of data processing. This target type can include at least one of the following: data addition, data deletion, data query, and data modification. Then, after determining the target type, the first processing process performs data processing on the target data based on that target type.

[0102] For example, when the target object is a session object in an instant messaging application, after determining the session connection descriptor corresponding to the session object, information is pushed to the corresponding session object in the instant messaging application based on the session connection descriptor; while when the target object is related information of media information in a media information application, such as comments on pictures or articles on a webpage, after determining the information content corresponding to the related information, such as the content of the comment, the corresponding related information in the media information application is queried or deleted based on the information content.

[0103] It should be noted that when the target type is data addition or deletion, while the first processing process performs data addition or deletion on the target data, it also updates the tree storage structure of the second processing process, thereby dynamically adjusting the tree storage structure of the second processing process. In this way, by dynamically adjusting the tree storage structure of the target data, the limitation of swoole_table's non-expandable memory is compensated for.

[0104] By applying the above embodiments of this application, a first processing process and a second processing process are set up. When the first processing process receives a data processing request, it determines the index information corresponding to the data processing request. Then, the second processing process determines the target data corresponding to the data processing request based on the index information. Finally, the first processing process performs target-type data processing on the target data. In this way, the first processing process processes the data processing request, while the second processing process stores and manages the shared data among the first processing processes. The combination of the two processing processes realizes data sharing and corresponding shared data processing, improving the shortcomings of the shared data storage structure and enhancing data storage performance.

[0105] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0106] Traditionally, there are two main solutions for inter-process data sharing in Swoole: one is to use Swoole's native `swoole_table`, and the other is to use external storage, such as databases like MySQL and MongoDB, or caching servers like Redis and Memcached. However, both methods have their own problems. Specifically, using Swoole's native `swoole_table` cannot store complex data structures. Furthermore, because the amount of storage needs to be pre-calculated during initialization and space allocation, and the table size cannot be modified during use, allocating too much space can lead to wasted memory, while allocating too little space may result in insufficient space for business applications. Using external storage requires a cross-machine network call, which is generally inefficient and impacts Swoole performance, making it unsuitable for performance-critical scenarios.

[0107] Based on this, embodiments of this application provide a data processing method, apparatus, device, computer-readable storage medium, and computer program product. Specifically, when the Swoole service starts, it sets up a task process (second processing process) and multiple worker processes (first processing process). The worker processes are used to handle user network requests, and the task process is used to store and manage shared data between workers. When a processing request is received for a target object (such as a session object in a session group or a comment on an article in a media message) in a target application (such as an instant messaging application or a media message application), the multiple worker processes receive the shared data between multiple workers stored by the task process by calling the interface provided by the task process. Based on the shared data, they perform CRUD operations on the target object to achieve the purpose of inter-process data sharing.

[0108] In practice, the server first detects user network requests (data processing requests). Upon detection, the server sends the request to the worker process, which parses it to determine the corresponding identification information (index information). For example, this identification information can be an application identifier. When the application is a session application, the identifier also includes a session group identifier and a user identifier; when the application is a web application, the identifier also includes article identifiers and comment identifiers on the webpage. After the worker process determines the identification information, it sends it to the task process. The task process then uses the index information to determine the corresponding shared data in its own storage data and sends the shared data back to the worker process, which then performs the corresponding data processing operations based on the shared data.

[0109] As an example, when the application is a session application, the identification information also includes the session group identifier and the user identifier. When the worker process sends the identification information to the task process, the task process indexes the information layer by layer in the tree-like storage structure with the application as the root node. Figure 4 As shown, specifically, the session application corresponding to the user's network request is first determined by the application identifier, then the session room in the session application corresponding to the user's network request is determined based on the session room identifier, then the session object in the session room is determined based on the user identifier, thereby determining the shared data stored corresponding to the session object, and sending the shared data to the worker process, which then performs corresponding data processing operations based on the shared data, such as pushing information to the session object.

[0110] It should be noted that the process of determining the shared data corresponding to the session object includes obtaining the mapping relationship between the session object and the shared data. After determining the session object, the shared data corresponding to the target user is determined based on the mapping relationship between the session object and the shared data.

[0111] In practice, before determining the shared data corresponding to the target user based on the task process, the shared data needs to be stored in the tree-structured storage of the task process. Specifically, first, the number of task processes is determined. When there is only one task process, all shared data is stored and maintained in that single task process. All worker processes interact with only one task process for data storage and retrieval. See details... Figure 6When multiple task processes exist, all shared data is stored and maintained in multiple different task processes. All worker processes interact with these task processes to store and retrieve shared data. Specifically, before storing shared data in a task process, the worker process performs a hash calculation on the shared data. Based on the hash value, it determines which task process the corresponding shared data should be stored in. This ensures that identical data is not stored in different task processes and also ensures accurate lookup during data retrieval. See details... Figure 7 .

[0112] It should be noted that when there are multiple task processes, the process of the worker process sending the identification information to the task process is as follows: based on the identification information, the identifier of the task process corresponding to the identification information is determined, and then based on the identifier, the identification information is sent to the corresponding task process.

[0113] In actual implementation, the data processing method provided in this application embodiment has two processing modes: synchronous data processing and asynchronous data processing.

[0114] For synchronous data processing, this method waits synchronously until the task completes or times out. The worker process must wait for the return or timeout before continuing with subsequent logic. See details. Figure 8 After a user sends a network data request, the worker process receives the request, parses it, and determines the corresponding identifier. This identifier is then sent to the task process, which uses it to identify the relevant shared data. The task process then returns this shared data to the worker process, which performs the corresponding data operation based on the shared data. After completing the operation, the worker process returns a completion message to the user, confirming the completion of the network request and allowing the user to initiate the next request. This synchronous processing model avoids deadlocks and dirty data reads, reducing the likelihood of errors during data processing.

[0115] For asynchronous data processing, this method does not require synchronous waiting; it returns immediately after the corresponding process completes its execution. That is, after the worker process sends the relevant information to the task process, the worker process can continue with subsequent logic without waiting for the task's processing result. See details... Figure 9After a user sends a network data request, the worker process receives the request, parses it to determine the corresponding identifier, and then sends this identifier to the task process. The task process then uses this identifier to determine the appropriate shared data to store. Simultaneously, the worker process can send a completion message to the user without waiting for the task's processing result, allowing the user to confirm the completion of the network request and initiate the next request. After the task process returns the corresponding shared data to the worker process, the worker process also performs the data operations for the corresponding network request based on this shared data.

[0116] It's worth noting that if you're concerned about the task's processing result, you can set an `onFinish` event callback. After the task is completed, the `onFinish` function will be called in the worker process, allowing for asynchronous processing of shared data. This asynchronous processing model improves data processing efficiency.

[0117] By applying the above embodiments of this application, a first processing process and a second processing process are set up. When the first processing process receives a data processing request, it determines the index information corresponding to the data processing request. Then, the second processing process determines the target data corresponding to the data processing request based on the index information. Finally, the first processing process performs target-type data processing on the target data. In this way, the first processing process processes the data processing request, while the second processing process stores and manages the shared data among the first processing processes. The combination of the two processing processes realizes data sharing and corresponding shared data processing, improving the shortcomings of the shared data storage structure and enhancing data storage performance.

[0118] The following description continues to illustrate the exemplary structure of the data processing apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the data processing device 455 of the memory 440 may include:

[0119] The first processing module 4551 is used to receive a data processing request for a target object, wherein the data processing request is used to request target data of the target object to be processed by the target type.

[0120] Response module 4552 is used to determine the index information corresponding to the target data in response to the data processing request;

[0121] The sending module 4553 is used to send the index information to the second processing process, and the second processing process is used to store the shared data of at least two first processing processes in a tree storage structure.

[0122] The second processing module 4554 is used to obtain the target data by indexing the tree storage structure through the second processing process based on the index information, and return the target data to the first processing process;

[0123] The execution module 4555 is used to perform data processing of the target type on the target data through the first processing process.

[0124] In some embodiments, the response module 4552 is further configured to respond to the data processing request by parsing the data processing request to obtain the object identifier of the target object and the node identifier of the parent index node to which the object identifier belongs when the object identifier is the index node; and to determine the object identifier and the node identifier as the index information corresponding to the target data.

[0125] In some embodiments, the second processing module 4554 is further configured to, when the data processing request is sent by an instant messaging application, the target object is a session object, and the parent index node is a session room, determine a tree storage structure with the instant messaging application as the root node through the second processing process; in the tree storage structure with the instant messaging application as the root node, index the session room to which the target object belongs based on the room identifier of the session room; index the session connection descriptor corresponding to the target object based on the session room and the object identifier of the session object, and use the session connection descriptor as the target data.

[0126] In some embodiments, the second processing module 4554 is further configured to, when the data processing request is sent by a media information application, the target object is the associated information of the media information, and the parent index node is the media information, determine a tree storage structure with the media information application as the root node through the second processing process; in the tree storage structure with the media information application as the root node, index the media information to which the target object belongs based on the media identifier of the media information; index the information content of the associated information based on the information identifier of the media information and the associated information, and use the information content of the associated information as the target data.

[0127] In some embodiments, the index information includes at least two levels. The second processing module 4554 is further configured to perform a layer-by-layer indexing of the tree storage structure based on the parent index information in the at least two levels of index information to determine the tree node corresponding to the parent index information; and to index the target data corresponding to the target object based on the tree node corresponding to the parent index information and the sub-index information in the at least two levels of index information.

[0128] In some embodiments, the execution module 4555 is further configured to obtain a target type for data processing of the target data; wherein the target type includes at least one of data addition, data deletion, data query, and data modification; and to perform data processing on the target data based on the target type.

[0129] In some embodiments, the apparatus further includes a first storage module, which is configured to: acquire the target data to be stored and the number of the second processing processes; when the number of the second processing processes is at least two, determine a second processing process for storing the target data; and store the target data in a tree storage structure of the determined second processing process.

[0130] In some embodiments, the first storage module is further configured to, when the number of the second processing processes is at least two, perform hash processing on the target data to obtain a processing result; obtain the identifier of each of the second processing processes; match the processing result with the identifier to obtain a matching result; and, based on the matching result, select a second processing process for storing the target data from at least two of the second processing processes.

[0131] In some embodiments, the apparatus further includes a second storage module, which is configured to: acquire the target data to be stored and the number of the second processing processes; when the number of the second processing processes is one, randomly generate an identifier corresponding to the second processing process; and store the target data in the tree storage structure of the second processing process based on the identifier.

[0132] In some embodiments, the number of the second processing processes is at least two, and the sending module 4553 is further configured to obtain the mapping relationship between the index information and each second processing process; determine the identifier of the second processing process corresponding to the index information based on the mapping relationship; and send the index information to the corresponding second processing process based on the identifier of the second processing process.

[0133] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described in this application.

[0134] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the data processing method provided in this application, for example... Figure 3 The data processing method is shown.

[0135] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0136] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0137] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0138] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0139] In summary, the embodiments of this application have the following beneficial effects:

[0140] (1) Data processing requests are processed by the first processing process, and the shared data between the first processing processes is stored and managed by the second processing process. The two processing processes are combined to realize data sharing and corresponding shared data processing, which improves the defects of the shared data storage structure and enhances the performance of data storage.

[0141] (2) The data processing process is realized through the asynchronous processing model, which improves the efficiency of data processing.

[0142] (3) Data processing is implemented through a synchronous processing model to avoid deadlock and dirty data reading, thus reducing the possibility of errors in the data processing process.

[0143] (4) The target data is stored through the tree storage structure of the second processing process, which makes up for the defect that swoole_table cannot store complex multidimensional data structures. At the same time, by storing different target data in the tree storage structure of different second processing processes, it is ensured that the target data can be accurately searched when reading data.

[0144] (5) By dynamically adjusting the tree storage structure of the target data, the defect of swoole_table's non-expandable memory is compensated.

[0145] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Multiple first processing processes and one or more second processing processes are configured, wherein the first processing processes are used to process user network requests; The first processing process receives a data processing request for a target object, the data processing request being used to request target data processing of the target type corresponding to the target object; In response to the data processing request, determine the index information corresponding to the target data; The index information is sent to the second processing process, wherein the second processing process is used to store shared data of at least two first processing processes in a tree storage structure, and the first processing process receives the shared data by calling the interface provided by the second processing process; When there are at least two second processing processes, the target data is hashed to obtain a processing result; based on the matching result of the processing result and the identifier of the second processing process, a second processing process for storing the target data is selected from at least two second processing processes. The target data is stored in the tree-structured storage of the determined second processing process; Based on the parent index information in at least two levels of index information, the tree storage structure is indexed layer by layer to determine the tree node corresponding to the parent index information; Based on the tree node corresponding to the parent index information and the sub-index information in the at least two levels of index information, the target data corresponding to the target object is obtained by indexing, and the target data is returned to the first processing process; The first processing process performs target-type data processing on the target data, wherein the first processing process employs a synchronous model to perform the following processing: waiting until the target data returned by the second processing process is received before performing target-type data processing on the target data; or... The first processing process uses an asynchronous model to perform the following processing: without waiting for the second processing process to return the target data, it continues to perform other subsequent processing, and after receiving the target data returned by the second processing process, it performs data processing of the target type on the target data.

2. The method as described in claim 1, characterized in that, The step of determining the index information corresponding to the target data in response to the data processing request includes: In response to the data processing request, the data processing request is parsed to obtain the object identifier of the target object and the node identifier of the parent index node to which the object identifier belongs when the object identifier is the index node; The object identifier and the node identifier are determined as the index information corresponding to the target data.

3. The method as described in claim 2, characterized in that, The method further includes: When the data processing request is sent by an instant messaging application, the target object is a session object, and the parent index node is a session room, the second processing process determines a tree storage structure with the instant messaging application as the root node. In a tree-structured storage system with the instant messaging application as the root node, the session room to which the target object belongs is indexed based on the room identifier of the session room. Based on the session room and the object identifier of the session object, the session connection descriptor corresponding to the target object is indexed and used as the target data.

4. The method as described in claim 2, characterized in that, The method further includes: When the data processing request is sent by a media information application, the target object is the associated information of the media information, and the parent index node is the media information, a tree storage structure with the media information application as the root node is determined through the second processing process. In a tree-structured storage system with the media information application as the root node, the media information to which the target object belongs is indexed based on the media identifier of the media information. Based on the information identifiers of the media information and the associated information, the information content of the associated information is indexed and used as the target data.

5. The method as described in claim 1, characterized in that, The step of performing the target type data processing on the target data through the first processing process includes: Obtain the target type for data processing of the target data; wherein, the target type includes at least one of data addition, data deletion, data query, and data modification; Based on the target type, the target data is processed.

6. The method as described in claim 1, characterized in that, Before performing a layer-by-layer indexing of the tree storage structure based on parent index information from at least two levels of index information to determine the tree node corresponding to the parent index information, the method further includes: Obtain the target data to be stored and the number of the second processing processes; When the number of the second processing processes is one, an identifier corresponding to the second processing process is randomly generated; Based on the identifier, the target data is stored in the tree storage structure of the second processing process.

7. The method as described in claim 1, characterized in that, The number of the second processing processes is at least two, and sending the index information to the second processing processes includes: Obtain the mapping relationship between the index information and each second processing process; Based on the mapping relationship, the identifier of the second processing process corresponding to the index information is determined; Based on the identifier of the second processing process, the index information is sent to the corresponding second processing process.

8. A data processing apparatus, characterized in that, The device includes: The first processing module is used to set up multiple first processing processes and one or more second processing processes, wherein the first processing processes are used to process user network requests; A data processing request for a target object is received, the data processing request being used to request target data of the target object to be processed according to the target type; The response module is used to determine the index information corresponding to the target data in response to the data processing request; A sending module is used to send the index information to the second processing process, wherein the second processing process is used to store shared data of at least two first processing processes in a tree storage structure, and the first processing process receives the shared data by calling the interface provided by the second processing process; The first storage module is configured to perform hash processing on the target data to obtain a processing result when the number of the second processing processes is at least two; select a second processing process for storing the target data from at least two second processing processes based on the matching result of the processing result and the identifier of the second processing process; and store the target data in the tree storage structure of the determined second processing process. The second processing module is used to perform a layer-by-layer indexing of the tree storage structure based on the parent index information in at least two levels of index information, so as to determine the tree node corresponding to the parent index information; Based on the tree node corresponding to the parent index information and the sub-index information in the at least two levels of index information, the target data corresponding to the target object is obtained by indexing, and the target data is returned to the first processing process; An execution module is configured to perform target-type data processing on the target data through the first processing process, wherein the first processing process employs a synchronous model to perform the following processing: waiting until the target data returned by the second processing process is received before performing target-type data processing on the target data; or, The first processing process uses an asynchronous model to perform the following processing: without waiting for the second processing process to return the target data, it continues to perform other subsequent processing, and after receiving the target data returned by the second processing process, it performs data processing of the target type on the target data.

9. The apparatus according to claim 8, characterized in that, The response module is further configured to respond to the data processing request by parsing the data processing request to obtain the object identifier of the target object and the node identifier of the parent index node to which the object identifier belongs when the object identifier is the index node. The object identifier and the node identifier are determined as the index information corresponding to the target data.

10. The apparatus according to claim 9, characterized in that, The second processing module is further configured to determine a tree storage structure with the instant messaging application as the root node through the second processing process when the data processing request is sent by an instant messaging application, the target object is a session object, and the parent index node is a session room. In a tree-structured storage system with the instant messaging application as the root node, the session room to which the target object belongs is indexed based on the room identifier of the session room. Based on the session room and the object identifier of the session object, the session connection descriptor corresponding to the target object is indexed and used as the target data.

11. The apparatus according to claim 9, characterized in that, The second processing module is further configured to determine a tree storage structure with the media information application as the root node through the second processing process when the data processing request is sent by the media information application, the target object is the associated information of the media information, and the parent index node is the media information. In a tree-structured storage system with the media information application as the root node, the media information to which the target object belongs is indexed based on the media identifier of the media information. Based on the information identifiers of the media information and the associated information, the information content of the associated information is indexed and used as the target data.

12. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the data processing method according to any one of claims 1 to 7.

13. A computer-readable storage medium, characterized in that, It stores executable instructions for causing a processor to execute, thereby implementing the data processing method according to any one of claims 1 to 7.

14. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the data processing method according to any one of claims 1 to 7.

Images