Data processing method and related equipment

Abstract

The invention discloses a data processing method and related equipment, and relates to the technical field of data processing, and the method comprises the steps: receiving target data sent by a client based on an interactive interface, the target data being target data sent by the client to a server based on a pre-configured interactive interface, the server further comprising a first configuration center, the first configuration center is the same as a second configuration center in the client, the first configuration center comprises multiple groups of configuration parameters, the interactive interface is pre-configured based on target configuration parameters, the interactive interface can be dynamically configured, and the target configuration parameters are determined from the multiple groups of configuration parameters based on the data type of target data; and processing the target data to obtain a processing result. According to the data processing method and device, the configuration of the interactive interface is dynamically adjusted based on the data type of the target data to be processed to receive the target data, that is, the target data of various data types can be processed through one interactive interface, and the data processing efficiency and portability are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and related equipment. Background Art

[0002] In the data processing world, there are a variety of file and message types, each of which typically requires a dedicated interface to ensure accurate data processing. However, this model has significant drawbacks. It requires developing and configuring individual interfaces for each data type, resulting in long development cycles and high error rates. Furthermore, the large number of interfaces and varying coding styles make troubleshooting, updating, and maintenance tedious, and constant adjustments and optimizations are required as data types change.

[0003] Therefore, how to process messages and files efficiently and conveniently is an issue that needs to be solved urgently.

[0004] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a data processing method and related equipment, aiming to solve the technical problem of how to process messages and files efficiently and conveniently.

[0006] To achieve the above objectives, the present application proposes a data processing method, which is applied to the server side. The data processing method includes:

[0007] receiving, based on the interactive interface, target data sent by the client, the target data being target data sent by the client to the server based on the pre-configured interactive interface, wherein the server further includes a first configuration center, the first configuration center being the same as the second configuration center in the client, the first configuration center including multiple sets of configuration parameters, the interactive interface being pre-configured based on the target configuration parameters, the interactive interface being dynamically configurable, the target configuration parameters being determined from the multiple sets of configuration parameters based on a data type of the target data;

[0008] The target data is processed to obtain a processing result.

[0009] In one embodiment, before the step of receiving the target data sent by the client based on the interactive interface, the method further includes:

[0010] receiving a data type sent by a client, where the data type is a data type determined by the client based on the target data;

[0011] Determining target configuration parameters from multiple groups of configuration parameters in a local first configuration center based on the data type;

[0012] The interactive interface is configured based on the target configuration parameters.

[0013] In one embodiment, the step of configuring the interactive interface based on the target configuration parameters further includes:

[0014] Obtaining a first configuration parameter currently corresponding to the interactive interface;

[0015] Determine whether the first configuration parameter is the same as the target configuration parameter; if so, directly obtain a configured interactive interface.

[0016] In one embodiment, the step of processing the target data to obtain a processing result further includes:

[0017] Determine the processing task corresponding to the target data and obtain local hardware resources;

[0018] Based on the hardware resources and the processing task, using a preset task processing algorithm to allocate the processing task into a plurality of subtasks;

[0019] Multiple threads are called to process the multiple subtasks in parallel to obtain data processing results.

[0020] To achieve the above objectives, the present application also proposes a data processing method, which is applied to a client. The data processing method includes:

[0021] Based on a preconfigured interactive interface, target data to be processed is sent to a server for the server to process the target data. The client further includes a second configuration center, which is the same as the first configuration center of the server. The second configuration center includes multiple groups of configuration parameters. The interactive interface is preconfigured based on the target configuration parameters. The interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple groups of configuration parameters based on the data type of the target data.

[0022] In one embodiment, before the step of sending the target data to be processed to the server based on the preconfigured interactive interface, the method further includes:

[0023] Acquire target data to be processed and determine the data type of the target data;

[0024] The data type is sent to the server side, so that the server side determines target configuration parameters based on the data type, and the target configuration parameters are used to configure the configuration interface of the server side.

[0025] In one embodiment, after the steps of obtaining the target data to be processed and determining the data type of the target data, the method further includes:

[0026] Determining target configuration parameters from multiple groups of configuration parameters in a local second configuration center based on the data type;

[0027] The interactive interface is configured based on the target configuration parameters.

[0028] In one embodiment, the step of configuring the interactive interface based on the target configuration parameters further includes:

[0029] Obtaining a second configuration parameter currently corresponding to the interactive interface;

[0030] Determine whether the second configuration parameter is the same as the target configuration parameter; if so, directly obtain the configured interactive interface.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, which is applied to a server side. The data processing device includes:

[0032] a receiving module, the receiving module being configured to receive target data sent by a client based on an interactive interface, the target data being target data sent by the client to a server based on the pre-configured interactive interface, wherein the server further includes a first configuration center, the first configuration center being the same as the second configuration center in the client, the first configuration center including multiple sets of configuration parameters, the interactive interface being pre-configured based on the target configuration parameters, the interactive interface being dynamically configurable, the target configuration parameters being determined from the multiple sets of configuration parameters based on a data type of the target data;

[0033] The data processing module is used to process the target data to obtain a processing result.

[0034] In one embodiment, the data processing device further includes a first configuration module, and the first configuration module includes:

[0035] a receiving unit, configured to receive a data type sent by a client, where the data type is a data type determined by the client based on the target data;

[0036] A first determining unit, configured to determine target configuration parameters from a plurality of groups of configuration parameters in a local first configuration center based on the data type;

[0037] A first configuration unit is configured to configure the interactive interface based on the target configuration parameters.

[0038] In one embodiment, the first configuration module further includes:

[0039] A first acquiring unit, configured to acquire a first configuration parameter currently corresponding to the interactive interface;

[0040] The first judgment unit is used to judge whether the first configuration parameter is the same as the target configuration parameter, and if so, directly obtain the configured interactive interface.

[0041] In one embodiment, the data processing module includes:

[0042] A second determining unit is configured to determine a processing task corresponding to the target data and obtain local hardware resources;

[0043] an allocating unit, configured to allocate the processing task into a plurality of subtasks based on the hardware resources and the processing task using a preset task processing algorithm;

[0044] The task processing unit is used to call multiple threads to process the multiple subtasks in parallel to obtain data processing results.

[0045] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, which is applied to a server side. The data processing device includes:

[0046] A sending module is used to send the target data to be processed to the server side based on a preconfigured interactive interface, so that the server side processes the target data. The client side also includes a second configuration center, which is the same as the first configuration center of the server side. The second configuration center contains multiple groups of configuration parameters. The interactive interface is preconfigured based on the target configuration parameters. The interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple groups of configuration parameters based on the data type of the target data.

[0047] In one embodiment, the data processing device further includes a determination module, and the determination module includes:

[0048] a second acquiring unit, configured to acquire target data to be processed and determine a data type of the target data;

[0049] The sending unit is configured to send the data type to the server side, so that the server side determines target configuration parameters based on the data type, and the target configuration parameters are used to configure the configuration interface of the server side.

[0050] In one embodiment, the data processing device further includes a second configuration module, wherein the second configuration module includes:

[0051] a third determining unit, configured to determine target configuration parameters from a plurality of groups of configuration parameters in a local second configuration center based on the data type;

[0052] The second configuration unit is configured to configure the interactive interface based on the target configuration parameters.

[0053] In one embodiment, the second configuration module further includes:

[0054] A third acquiring unit, configured to acquire a second configuration parameter currently corresponding to the interactive interface;

[0055] The second judgment unit is used to judge whether the second configuration parameter is the same as the target configuration parameter, and if so, directly obtain the configured interactive interface.

[0056] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0057] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0058] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the data processing method described above.

[0059] One or more technical solutions proposed in this application have at least the following technical effects:

[0060] The present application proposes a data processing method and related equipment, which relate to the field of data processing technology. In the field of data processing, there are various types of files and messages, and each type usually requires a dedicated interface to ensure accurate data processing. However, this model has significant disadvantages. It is necessary to develop and configure interfaces for each data type one by one, which has a long development cycle and is prone to errors. In addition, there are many interfaces and different code styles, which makes troubleshooting and updating maintenance cumbersome. In addition, as the data type changes, continuous adjustment and optimization are required. In this application, first, based on the interactive interface, target data sent by the client is received. The target data is the target data sent by the client to the server based on the pre-configured interactive interface. The server also includes a first configuration center, which is the same as the second configuration center in the client. The first configuration center contains multiple groups of configuration parameters. The interactive interface is pre-configured based on the target configuration parameters. The interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple groups of configuration parameters based on the data type of the target data. Then, the target data is processed to obtain a processing result.

[0061] It can be understood that the present application dynamically adjusts the configuration of the interactive interface based on the data type of the target data to be processed to receive the target data. That is, target data of various data types can be processed through one interactive interface, thereby improving the efficiency and portability of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0064] Figure 1 A flowchart of the first embodiment of the data processing method of this application is provided;

[0065] Figure 2 A schematic diagram of the asynchronous-to-synchronous conversion process provided in Example 1 of the data processing method of this application;

[0066] Figure 3 A flowchart of the second embodiment of the data processing method of this application is provided;

[0067] Figure 4 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;

[0068] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0069] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0070] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0071] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0072] The main solutions of the embodiments of this application are:

[0073] In this embodiment, for ease of description, the following description is made with the data processing device as the execution subject.

[0074] Due to existing technology, in the data processing field, there are many different file and message types, and each type usually requires a dedicated interface to ensure accurate data processing. However, this model has significant drawbacks. It requires the development and configuration of interfaces for each data type, which is long and error-prone. The large number of interfaces and different coding styles make troubleshooting and updating and maintenance tedious, and continuous adjustment and optimization are required as the data type changes.

[0075] The present application provides a solution, which includes: first, receiving target data sent by a client based on an interactive interface, wherein the target data is target data sent by the client to a server based on the pre-configured interactive interface, wherein the server also includes a first configuration center, which is the same as the second configuration center in the client, and includes multiple groups of configuration parameters. The interactive interface is pre-configured based on the target configuration parameters, and the interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple groups of configuration parameters based on the data type of the target data; and then, the target data is processed to obtain a processing result.

[0076] It can be understood that the present application dynamically adjusts the configuration of the interactive interface based on the data type of the target data to be processed to receive the target data. That is, target data of various data types can be processed through one interactive interface, thereby improving the efficiency and portability of data processing.

[0077] It should be noted that the execution subject of this embodiment can be a computing service device with network communication and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or data processing device capable of implementing the above functions. The following uses a data processing device as an example to illustrate this embodiment and the following embodiments.

[0078] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data processing method of this application.

[0079] In this embodiment, applied to the server side, the data processing method includes steps S100 to S200:

[0080] Step S100: Receive target data sent by a client based on an interactive interface. The target data is target data sent by the client to a server based on a pre-configured interactive interface. The server further includes a first configuration center, which is the same as a second configuration center in the client. The first configuration center includes multiple sets of configuration parameters. The interactive interface is pre-configured based on the target configuration parameters. The interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple sets of configuration parameters based on the data type of the target data.

[0081] It should be noted that the execution subject of this embodiment can be a computing service device with network communication and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or data processing device capable of implementing the above functions. The following uses a data processing device as an example to illustrate this embodiment and the following embodiments.

[0082] It should be noted that in this application, there is at least one predefined interactive interface between the server and the client, and the client will send the collected, organized or generated target data with specific purposes to the server through this interface.

[0083] It should be noted that the target data may be message data, file data, etc.

[0084] It should be noted that for different types of message data or file data, the corresponding interactive interfaces and transmission protocols are different. For example, for API requests or responses in JSON or XML format (structured data, such as user information, order data), the corresponding interactive interface is the RESTful API interface or the GraphQL interface, and the corresponding transmission protocol is the HTTP protocol or the HTTPS protocol. If the target data is file data, such as video file data, image file data, compressed package data, or backup file data, the corresponding interactive interface is the FTP / SFTP client interface or the cloud storage service API interface, and the corresponding transmission protocol is the FTP protocol / SFTP protocol.

[0085] Specifically, for TCP clients, there are mainly the following message reading methods:

[0086] The first message reading method, length-field message reading, is used when sending and reading messages to clients in the format of {n-bit length field} + {message content}. This is called length-field message reading. For example, if you want to send "Today is March 1, 2025, and the weather is great today," and use the UTF-8 encoding format with a 6-bit length field, the message will be converted to a byte array with a length of 45, and the final message sent will be: 000045 Today is March 1, 2025, and the weather is great today. This means that the 6-bit length field at the beginning indicates the true length of the message body. The same is true for reading messages: the first 6 bits are first received to determine the guaranteed overall length, and then the remaining portion of the message is read for the specified length. There are many variations of length-field messages, such as those with fixed values ​​before and after the length field or message-specific notations. These are collectively referred to as length-field mode. To read messages using length-field mode, simply use readerType = HEAD_LENGTH.

[0087] The second message reading method is fixed-end message reading. Fixed-end message reading means that, as agreed with the customer, the message ends with a fixed string. For example, in the above configuration, the readerType tag is configured as FIX_END, indicating fixed-end message reading. The fixedEnd tag also needs to be configured. For example, suppose the returned message string is "Today is March 1, 2025. The weather is very good today#." The # here indicates that the returned message is complete and no further message reading is required.

[0088] The third message reading method is fixed-length message reading: a fixed-length message indicates that the message is 50 bits long, and the data processing device only needs to read 50 bits to complete the message reading. If the message length exceeds 50, the data processing device will only return 50 bits. If the message length is less than 50, the data processing device will continue reading until the input stream is closed.

[0089] The fourth message reading method is to read messages without any clear characteristics. Messages without any clear characteristics are those that, during transmission to the client, lack a length field, a fixed end, or even a fixed length. It is difficult to clearly determine the end of such messages, so data processing equipment has designed a message reader to address this problem.

[0090] The fifth message reading method is the customized message reading method: If the message is highly personalized, it can be completed through simple rewriting. The data processing device encapsulates the client and shields the communication layer. Only the input stream needs to be rewritten.

[0091] It should be noted that the client and server have a high degree of consistency and coordination in configuration, which can ensure that both parties follow the same rules and framework during data interaction.

[0092] It should be noted that the interactive interface is pre-configured based on target configuration parameters, but also has the ability to be dynamically configured. This design allows the interactive interface to be flexibly adjusted according to actual needs to adapt to different types of data transmission and different business scenarios.

[0093] It should be noted that the target configuration parameters are determined from multiple sets of configuration parameters contained in the first configuration center, and are determined based on the data type of the target data. The system can select the most appropriate configuration parameters to set up the interactive interface based on the characteristics of different data types, thereby optimizing the efficiency and quality of data transmission.

[0094] For example, consider an online shopping system where the client is a mobile shopping app used by users, and the server handles various business logic and data storage. During the user order process, the client sends the user's order information (target data) to the server for processing.

[0095] To ensure that data can be transmitted accurately, an order-related interactive interface is pre-set in the system. The configuration of this interface is based on the parameters in the first configuration center on the server side.

[0096] Moreover, this interface can dynamically adjust its configuration parameters according to different situations of order data (such as whether it is a message type order or a file type order) to better adapt to the data transmission requirements of various order types. These adjustments are achieved by selecting the target configuration parameters corresponding to the order data type from multiple sets of configuration parameters in the first configuration center.

[0097] Specifically, before the step of receiving the target data sent by the client based on the interactive interface, steps S10 to S30 are also included:

[0098] Step S10, receiving a data type sent by the client, where the data type is a data type determined by the client based on the target data;

[0099] Step S20: determining target configuration parameters from multiple groups of configuration parameters in the local first configuration center based on the data type;

[0100] It should be noted that, in this application, multiple groups of configuration parameters are set in advance. The configuration parameters can be custom configuration parameters or regular configuration parameters. The configuration parameters can be encapsulation protocols.

[0101] As you can see, configuring interactive interfaces through encapsulated protocols avoids the lengthy code writing required in traditional development models. This design not only significantly shortens the development cycle, enabling developers to quickly enter the joint debugging and testing phase, but also significantly reduces subsequent maintenance costs. Because the code is encapsulated and hidden, developers no longer need to troubleshoot code line by line, allowing them to focus more on implementing the interface's business functions.

[0102] It is understandable that different data types have different corresponding interface protocols.

[0103] For example, if the data type is TCP message, the target configuration parameters are: id (configuration ID), Type (type, fill in TCP for TCP server), port (local port opened as TCP server), backLog (maximum number of connections created by the TCP server at one time), acceptTimeoutMs (timeout in milliseconds for creating a socket, that is, the timeout for listening for a connection at one time. If it exceeds 5000, a debug log will be printed), taskCoreThreadNum (the number of core threads of the processing thread pool after receiving the message, set as needed), taskMaxThreadNum (the maximum number of threads of the processing thread pool after receiving the message, set as needed), taskBlockQueueNum (the number of blocking queues of the processing thread pool after receiving the message, set as needed), readerType (message reading type, supports HEAD_LENGTH type: length field reading method, combined with lengthOffset tag method; NETWORK_LONG_INTEGER type: network sequence length integer, the default first 4 bytes are the length field size; NETWORK_SHORT_INTEGER type: network sequence short integer). Integer, by default the first two bytes are the length field size; FIX_END type: fixed string end method, requires the configuration of the fixedEnd tag; FIX_LENGTH type: fixed-length message, requires the configuration of the fixedLength tag; ALL_BYTES type: complete message reading method), lengthOffset (length field, valid when readerType = HEAD_LENGTH. A TCP request message consists of {n-bit length field} + {message body corresponding to the length field number}), lengthPaddingType (length field padding method, supports left-padded with zeros, left-padded with spaces, and right-padded with spaces, with configuration options of LEFT_ZERO, LEFT_SPACE, and RIGHT_SPACE), lengthCalculate (whether the length field calculation includes the length field itself; by default it does not. If included, lengthOffset is added to the officially sent message as the length field value), fixedEnd (fixed end of message reading, valid when readerType = FIX_END), and fixedLength (fixed-length message reading method, valid when readerType = FIX_LENGTH).

[0104] For example, if the data type is MQ message, the target configuration parameter may be a RabbitMQ listening queue configuration parameter or an IBMMQ listening queue configuration parameter.

[0105] Specifically, the RabbitMQ listening queue is configured as follows: label (description), type (fill in MQ on the RabbitMQ server), exchange (switch), bindKey (binding key, i.e., routing key for binding the switch and queue), queue (listening queue name), exchangeType (switch type, default is direct, can be changed to fanout or topic), taskCoreThreadNum (the number of core threads of the processing thread pool after receiving the message, defaults to 5, set as needed), taskMaxThreadNum (the maximum number of threads of the processing thread pool after receiving the message, defaults to 10, set as needed) and taskBlockQueueNum (the number of blocking queues of the processing thread pool after receiving the message, defaults to 128, set as needed).

[0106] Specifically, the IBMMQ listening queue is configured as follows: label (description), type (fill in ibmmq on the ibmmq server side), queueManager (queue manager name), queueName (name of the receiving message queue), notifyListener (default false, whether to notify the listener, true indicates notification listening mode, that is, to enable listening by calling the notification listener, which can be used when the request volume is not large. After sending the request, the listening of the receiving return queue is enabled. false indicates the listening mode, that is, to always listen for messages in the message return queue), listenTimeoutMs (default 20000, the number of milliseconds for a message to be monitored at a time. If it is in the listening mode, the debug log will be printed after one monitoring), connectOptions (the connectOptions parameter of the queue manager when it is created. When configuration is required, it supports numbers or MQC interface attribute configuration. Multiple can be separated by "|", for example: 1|2 or MQC.MQGMO_VERSION_1| MQC.MQGMO_VERSION_2 are both acceptable), openOptionsArg (parameters of the queue when initializing the connection, support digital or MQC interface property configuration, default 8193, that is: 1|8192, or MQC.MQOO_INPUT_AS_Q_DEF|MQC.MQOO_FAIL_IF_QUIESCING, if you need to change, please configure the digital or MQC property), gmoOptions (initialization configuration in the message receiving options, that is, options configuration in MQGetMessageOptions, support digital or MQC interface property configuration, support initialization configuration and interceptor call configuration), taskCoreThreadNum (default 5, the number of core processing threads of the thread pool after receiving the message), taskMaxThreadNum (default 10, the maximum number of processing threads of the thread pool after receiving the message) and taskBlockQueueNum (default 128, the maximum length of the waiting queue of the thread pool after receiving the message).

[0107] Step S30: configuring the interactive interface based on the target configuration parameters.

[0108] Specifically, the step of configuring the interactive interface based on the target configuration parameters further includes steps S31 to S32:

[0109] Step S31, obtaining a first configuration parameter currently corresponding to the interactive interface;

[0110] It should be noted that the first configuration parameter refers to the configuration parameter corresponding to the data processed last time.

[0111] Step S32: determine whether the first configuration parameter is the same as the target configuration parameter; if they are the same, directly obtain the configured interactive interface.

[0112] It is understandable that if the data type corresponding to the data processed last time is the same as the data type corresponding to the data processed this time, the interaction interface configured last time can be directly used without reconfiguring the interaction interface.

[0113] Step S200: Process the target data to obtain a processing result.

[0114] Specifically, the step of processing the target data to obtain a processing result further includes steps S210 to S230:

[0115] Step S210, determining the processing task corresponding to the target data and obtaining local hardware resources;

[0116] In this example, we first need to clearly define the processing operations to be performed on the target data. For example, if the target data is a piece of text, the processing tasks could be text translation, sentiment analysis, grammar checking, etc. Only after the processing tasks are clearly defined can we proceed to the next steps, such as resource allocation and task decomposition.

[0117] Furthermore, after determining the processing task, it is necessary to fully understand the available local hardware resources. Local hardware resources typically include the central processing unit (CPU), graphics processing unit (GPU), random access memory (RAM), storage devices (such as hard drives and solid-state drives), etc. For example, when performing image processing tasks, it is necessary to understand whether the local computer's GPU supports relevant image processing acceleration functions, the number of CPU cores (which affects the number of threads that can simultaneously process tasks), the RAM capacity (which affects the amount of data that can be processed simultaneously), and the read and write speed of the storage device (which affects the efficiency of data input and output).

[0118] Step S220, based on the hardware resources and the processing task, using a preset task processing algorithm to allocate the processing task into a plurality of subtasks;

[0119] After clarifying the processing task and understanding the local hardware resources, it's important to comprehensively consider these two factors when allocating tasks. For example, if the task involves complex computations involving large amounts of data, and the local system has a high-performance multi-core CPU and large amounts of RAM, consider breaking the task down into multiple subtasks suitable for multi-core processing. These subtasks can be processed in parallel using the CPU's multiple cores, while the large amount of RAM ensures sufficient memory space for temporary data storage during the processing of these subtasks.

[0120] Furthermore, a preset task processing algorithm is used to distribute the processing task into multiple subtasks: This involves a pre-set task processing algorithm. The function of this algorithm is to reasonably split the original complete processing task into multiple relatively small subtasks based on the characteristics of the task and the status of the hardware resources. For example, in a text semantic analysis task, the preset task processing algorithm may first segment the text, and then distribute the entire text analysis task into multiple subtasks such as part-of-speech tagging and syntactic analysis for different semantic units based on the semantic units after segmentation. This allows different subtasks to be processed simultaneously on different hardware resources (such as different CPU cores), thereby improving the overall processing efficiency.

[0121] Step S230: Call multiple threads to process the multiple subtasks in parallel to obtain data processing results.

[0122] It's important to note that a thread is the fundamental unit of computer program execution. In an operating system, a thread is the smallest unit for computational scheduling. Calling multiple threads leverages the operating system's multithreading mechanism, allowing multiple subtasks to be processed simultaneously on different threads. For example, in a multi-core CPU environment, each subtask can run in a separate thread, and these threads can be assigned to different CPU cores, enabling parallel processing of multiple subtasks. This fully utilizes the parallel processing capabilities of hardware resources, significantly reducing overall data processing time.

[0123] When all subtasks processed in parallel by multiple threads are completed, the processing results of these subtasks are summarized and integrated, and finally the processing result of the entire target data can be obtained. This result is the final output of the entire data processing flow and can be used for subsequent use or storage operations.

[0124] In addition, it should be noted that the reference Figure 2In complex application interactions, the transition between asynchronous and synchronous processing may occur at any time. For example, the client system may provide an asynchronous interface, but the data processing system or upstream requires a synchronous return. This involves converting asynchronous data to synchronous data. Through simple configuration, the front-end SDK provides asynchronous-to-synchronous conversion solutions for both single-instance and multi-instance deployments. In a single-instance deployment, the thread wait and wake-up mechanism is used to provide the asynchronously returned data to the synchronously waiting thread and return it. In a multi-instance deployment, the asynchronous return may occur on the same instance or different instances. The single-instance thread wait and wake-up mechanism, combined with Redis middleware and databases (configurable options), ensures that the waiting thread can obtain the asynchronously returned data. The synchronously waiting thread will sleep. If the asynchronously returned data is on the same instance, the waiting thread will be directly awakened. If the asynchronously returned data is not on the same instance, the waiting thread will automatically wake up to query and obtain the asynchronous return result.

[0125] The present application proposes a data processing method and related equipment, which relate to the field of data processing technology. In the field of data processing, there are various types of files and messages, and each type usually requires a dedicated interface to ensure accurate data processing. However, this model has significant disadvantages. It is necessary to develop and configure interfaces for each data type one by one, which has a long development cycle and is prone to errors. In addition, there are many interfaces and different code styles, which makes troubleshooting and updating maintenance cumbersome. In addition, as the data type changes, continuous adjustment and optimization are required. In this application, first, based on the interactive interface, target data sent by the client is received. The target data is the target data sent by the client to the server based on the pre-configured interactive interface. The server also includes a first configuration center, which is the same as the second configuration center in the client. The first configuration center contains multiple groups of configuration parameters. The interactive interface is pre-configured based on the target configuration parameters. The interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple groups of configuration parameters based on the data type of the target data. Then, the target data is processed to obtain a processing result.

[0126] It can be understood that the present application dynamically adjusts the configuration of the interactive interface based on the data type of the target data to be processed to receive the target data. That is, target data of various data types can be processed through one interactive interface, thereby improving the efficiency and portability of data processing.

[0127] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 3 , applied to the client, the data processing method further includes step A100:

[0128] Step A100: Based on a pre-configured interactive interface, the target data to be processed is sent to the server side for the server side to process the target data, wherein the client side also includes a second configuration center, which is the same as the first configuration center of the server side, and the second configuration center contains multiple groups of configuration parameters. The interactive interface is pre-configured based on the target configuration parameters, and the interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple groups of configuration parameters based on the data type of the target data.

[0129] Specifically, before the step of sending the target data to be processed to the server based on the pre-configured interactive interface, steps A10 to A20 are also included:

[0130] Step A10, obtaining target data to be processed and determining the data type of the target data;

[0131] Step A20: Send the data type to the server side, so that the server side determines target configuration parameters based on the data type, and the target configuration parameters are used to configure the configuration interface of the server side.

[0132] Specifically, after the steps of obtaining the target data to be processed and determining the data type of the target data, steps B10 to B20 are further included:

[0133] Step B10: determining target configuration parameters from multiple groups of configuration parameters in the local second configuration center based on the data type;

[0134] Step B20: configuring the interactive interface based on the target configuration parameters.

[0135] For example, if the data type is TCP message, the target configuration parameters are: id (unique identification number of each client / server), encoding (encoding, supports UTF8, UTF16, GB2312, GBK, GB18030, ISO_8859_1), ip (IP of the requested TCP server), Port (port of the requested TCP server), connectTimeoutMs (connection time in milliseconds for the TCP server, the minimum value is 200, if it is less than 200, the default is 200), readTimeoutMs (TCP read timeout in milliseconds, the minimum value is 500), readerType (message reading type, supports HEAD_LENGTH type: length field reading method, with length thOffset tag mode; NETWORK_LONG_INTEGER type: network sequence length integer, the default first 4 bytes are the length field size; NETWORK_SHORT_INTEGER type: network sequence short integer, the default first 2 bytes are the length field size; FIX_END type: fixed string end method, you need to configure the fixedEnd tag; FIX_LENGTH type: fixed length message, you need to configure the fixedLength tag; ALL_BYTES type: complete message reading method, you need to pay attention to the business scenario when using it to avoid long-term connection occupation, reading until the input stream is closed or timed out), lengthOffset (length field, readerType = HEAD_LENGTH is valid.A TCP request message is {n-bit length field} + {message body corresponding to the length field number}), lengthPaddingType (length field filling method, supports left-padded 0 / left-padded space / right-padded space, the configuration items are LEFT_ZERO, LEFT_SPACE, RIGHT_SPACE), lengthCalculate (whether the length field calculation includes the length of the length field itself, not included by default. If included, lengthOffset will be added to the officially sent message as the value of the length field), fixedEnd (fixed end of message reading, readerType=FIX_END is valid), fixedLength (fixed-length message reading method, readerType=FIX_LENGTH is valid), shutdownOutput (whether to shut down Output stream, when set to true, the output stream will be closed directly after the message is sent, without affecting the TCP connection), respType (receiving method of the return message, DIRECT - synchronous reception of messages, default; SERVER - receiving messages through other servers, that is, asynchronous to synchronous solution), syncServerId (configId of the synchronous message receiving server, required when respType=SERVER), syncWaitMillis (milliseconds of synchronous waiting, valid when respType=SERVER) and wakeUpMills (self-wake-up milliseconds, when the mode of receiving messages through other servers is turned on, after successfully sending the message, it will sleep during this period, waiting for other threads to wake up or self-wake up to check whether there is a return message, valid when respType=SERVER).

[0136] For example, if the data type is file, the target configuration parameter is an FTP file transfer client configuration parameter or an SFTP file transfer client configuration parameter.

[0137] Specifically, the FTP file transfer client configuration parameters are: label (description), Type (fill in ftp for ftp client type), host (address of ftp server), mode (file transfer mode, active mode: ACTIVE; passive mode: PASSIVE, the default is passive mode), username (login username of ftp server), password (login password of ftp server), connectTimeout (connection timeout, in milliseconds, the default is 10 seconds) and ftpPath (path on the file server).

[0138] Specifically, the SFTP file transfer client configuration parameters are: label (description), type (fill in sftp for the sftp client), host (address of the sftp server), username (user name of the server, supports password-free login configuration, uses identityPath+passphrase configuration), port (port), password (server password, supports password-free login configuration, uses identityPath+passphrase configuration), knowHosts() knowHosts configuration, file address, if not configured, the default non-verification method will be used), identityPath (certificate path address), passphrase (certificate password) and ftpPath (path on the file server).

[0139] Specifically, the step of configuring the interactive interface based on the target configuration parameters further includes steps B21 to B22:

[0140] Step B21, obtaining a second configuration parameter currently corresponding to the interactive interface;

[0141] It should be noted that the second configuration parameter refers to the configuration parameter corresponding to the data processed last time.

[0142] Step B22: determine whether the second configuration parameter is the same as the target configuration parameter; if so, directly obtain the configured interactive interface.

[0143] It is understandable that if the data type corresponding to the data processed last time is the same as the data type corresponding to the data processed this time, the interaction interface configured last time can be directly used without reconfiguring the interaction interface.

[0144] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0145] This application also proposes a data processing device, please refer to Figure 4 , applied to the server side, the data processing device includes:

[0146] A receiving module 10 is configured to receive target data sent by a client based on an interactive interface. The target data is target data sent by the client to a server based on a pre-configured interactive interface. The server further includes a first configuration center, which is the same as the second configuration center in the client. The first configuration center includes multiple sets of configuration parameters. The interactive interface is pre-configured based on the target configuration parameters. The interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple sets of configuration parameters based on the data type of the target data.

[0147] The data processing module 20 is used to process the target data to obtain a processing result.

[0148] In one embodiment, the data processing device further includes a first configuration module, and the first configuration module includes:

[0149] a receiving unit, configured to receive a data type sent by a client, where the data type is a data type determined by the client based on the target data;

[0150] A first determining unit, configured to determine target configuration parameters from a plurality of groups of configuration parameters in a local first configuration center based on the data type;

[0151] A first configuration unit is configured to configure the interactive interface based on the target configuration parameters.

[0152] In one embodiment, the first configuration module further includes:

[0153] A first acquiring unit, configured to acquire a first configuration parameter currently corresponding to the interactive interface;

[0154] The first judgment unit is used to judge whether the first configuration parameter is the same as the target configuration parameter, and if so, directly obtain the configured interactive interface.

[0155] In one embodiment, the data processing module includes:

[0156] A second determining unit is configured to determine a processing task corresponding to the target data and obtain local hardware resources;

[0157] an allocating unit, configured to allocate the processing task into a plurality of subtasks based on the hardware resources and the processing task using a preset task processing algorithm;

[0158] The task processing unit is used to call multiple threads to process the multiple subtasks in parallel to obtain data processing results.

[0159] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, which is applied to a server side. The data processing device includes:

[0160] A sending module is used to send the target data to be processed to the server side based on a preconfigured interactive interface, so that the server side processes the target data. The client side also includes a second configuration center, which is the same as the first configuration center of the server side. The second configuration center contains multiple groups of configuration parameters. The interactive interface is preconfigured based on the target configuration parameters. The interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple groups of configuration parameters based on the data type of the target data.

[0161] In one embodiment, the data processing device further includes a determination module, and the determination module includes:

[0162] a second acquiring unit, configured to acquire target data to be processed and determine a data type of the target data;

[0163] The sending unit is configured to send the data type to the server side, so that the server side determines target configuration parameters based on the data type, and the target configuration parameters are used to configure the configuration interface of the server side.

[0164] In one embodiment, the data processing device further includes a second configuration module, wherein the second configuration module includes:

[0165] a third determining unit, configured to determine target configuration parameters from a plurality of groups of configuration parameters in a local second configuration center based on the data type;

[0166] The second configuration unit is configured to configure the interactive interface based on the target configuration parameters.

[0167] In one embodiment, the second configuration module further includes:

[0168] A third acquiring unit, configured to acquire a second configuration parameter currently corresponding to the interactive interface;

[0169] The second judgment unit is used to judge whether the second configuration parameter is the same as the target configuration parameter, and if so, directly obtain the configured interactive interface.

[0170] The data processing device provided in this application utilizes the data processing method of the above-described embodiments to solve the technical problems of data processing. Compared with the prior art, the beneficial effects of the data processing device provided in this application are the same as those of the data processing method provided in the above-described embodiments. Other technical features of the data processing device are the same as those disclosed in the above-described embodiments and are not further described here.

[0171] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data processing method in the above-mentioned embodiment 1.

[0172] Reference below Figure 5 , which shows a schematic diagram of the structure of a data processing device suitable for implementing the embodiments of the present application. The data processing device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The data processing device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0173] like Figure 5As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the data processing device. Processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. Communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0174] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0175] The data processing device provided by this application, using the data processing method in the above-mentioned embodiment, can solve the technical problems. Compared with the prior art, the beneficial effects of the data processing device provided by this application are the same as those of the data processing method provided by the above-mentioned embodiment, and the other technical features of the data processing device are the same as those disclosed in the method in the above-mentioned embodiment, and are not further described here.

[0176] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0177] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0178] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0179] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0180] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0181] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a data processing device, the data processing device:

[0182] receiving, based on the interactive interface, target data sent by the client, the target data being target data sent by the client to the server based on the pre-configured interactive interface, wherein the server further includes a first configuration center, the first configuration center being the same as the second configuration center in the client, the first configuration center including multiple sets of configuration parameters, the interactive interface being pre-configured based on the target configuration parameters, the interactive interface being dynamically configurable, the target configuration parameters being determined from the multiple sets of configuration parameters based on a data type of the target data;

[0183] The target data is processed to obtain a processing result.

[0184] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0185] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0186] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0187] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data processing method, thereby solving technical problems in data processing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data processing method provided in the above-described embodiment, and are not further elaborated here.

[0188] The present application also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0189] The computer program product provided in this application can solve the technical problem of data processing. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the data processing method provided in the above embodiment, which will not be repeated here.

[0190] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, applied to a server, characterized in that: The data processing method includes: receiving, based on the interactive interface, target data sent by the client, the target data being target data sent by the client to the server based on the pre-configured interactive interface, wherein the server further includes a first configuration center, the first configuration center being the same as the second configuration center in the client, the first configuration center including multiple sets of configuration parameters, the interactive interface being pre-configured based on the target configuration parameters, the interactive interface being dynamically configurable, the target configuration parameters being determined from the multiple sets of configuration parameters based on a data type of the target data; The target data is processed to obtain a processing result.

2. The data processing method according to claim 1, wherein: Before the step of receiving the target data sent by the client based on the interactive interface, the method further includes: receiving a data type sent by a client, where the data type is a data type determined by the client based on the target data; Determining target configuration parameters from multiple groups of configuration parameters in a local first configuration center based on the data type; The interactive interface is configured based on the target configuration parameters.

3. The data processing method according to claim 2, wherein: The step of configuring the interactive interface based on the target configuration parameters further includes: Obtaining a first configuration parameter currently corresponding to the interactive interface; It is determined whether the first configuration parameter is the same as the target configuration parameter. If they are the same, a configured interactive interface is directly obtained.

4. The data processing method according to claim 1, wherein: The step of processing the target data to obtain a processing result further includes: Determine the processing task corresponding to the target data and obtain local hardware resources; Based on the hardware resources and the processing task, using a preset task processing algorithm to allocate the processing task into a plurality of subtasks; Multiple threads are called to process the multiple subtasks in parallel to obtain data processing results.

5. A data processing method, applied to a client, characterized in that: The data processing method includes: Based on a preconfigured interactive interface, target data to be processed is sent to a server for the server to process the target data. The client further includes a second configuration center, which is the same as the first configuration center of the server. The second configuration center includes multiple groups of configuration parameters. The interactive interface is preconfigured based on the target configuration parameters. The interactive interface can be dynamically configured. The target configuration parameters are determined from the multiple groups of configuration parameters based on the data type of the target data.

6. The data processing method according to claim 5, wherein: Before the step of sending the target data to be processed to the server based on the pre-configured interactive interface, the method further includes: Acquire target data to be processed and determine the data type of the target data; The data type is sent to the server side, so that the server side determines target configuration parameters based on the data type, and the target configuration parameters are used to configure the configuration interface of the server side.

7. The data processing method according to claim 6, wherein: After the steps of obtaining the target data to be processed and determining the data type of the target data, the method further includes: Determining target configuration parameters from multiple groups of configuration parameters in a local second configuration center based on the data type; The interactive interface is configured based on the target configuration parameters.

8. The data processing method according to claim 7, wherein: The step of configuring the interactive interface based on the target configuration parameters further includes: Obtaining a second configuration parameter currently corresponding to the interactive interface; Determine whether the second configuration parameter is the same as the target configuration parameter; if so, directly obtain the configured interactive interface.

9. A data processing device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 4, or any one of 5 to 8.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 4 or any one of claims 5 to 8 are implemented.

Images