A method of data processing and related apparatus

Abstract

This application discloses a data processing method and related apparatus that dynamically adjusts the data format of business data in shared memory, solving the problem of data format mismatch caused by adding business data and improving the user experience of the target object. The method includes obtaining a first data description file, which describes the data format of first business data, the first business data being business data obtained when the loading service program is in a first state; when the version of the first data description file is higher than the version of a second data description file, determining data difference information between the first and second data description files, the second data description file describing the data format of second business data stored in shared memory, the second business data being business data obtained before the loading service program was in the first state; and adjusting the data format of the second business data based on the data difference information and the data format of the first business data.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data processing method and related apparatus. Background Technology

[0002] Shared memory is an efficient inter-process communication method. During the execution of a service program, data is written to shared memory, enabling persistent data storage instead of writing to local disks or remote databases. This allows the service program to be quickly restarted in case of an exception, providing necessary services to one or more applications without affecting the user experience. Furthermore, it reduces the loss of unsynchronized data during exceptions.

[0003] In the relevant solutions, when the service program starts for the first time, it needs to request memory space from the system and divide the requested memory space into a grid. This way, when the service program starts subsequently, it checks whether the shared memory associated with the service program exists, and if so, writes the data into the shared memory. It also checks whether the grid size of the shared memory matches the size of the service program's business data. However, since shared memory is generally a fixed size, only a fixed length of shared memory can be requested, and the requested shared memory cannot be dynamically increased. Furthermore, the size of business data nodes is fixed, and adjusting business data during online operation is difficult. The only solution is to reserve extra space in shared memory and use this reserved space to cache business data. However, this solution of reserving extra space in shared memory to store business data requires strictly maintaining the original data format of the business data before adjustment, and new business data can only be added after the original data. This leads to data format mismatch issues when dynamically adding business data during daily operation, thus affecting the user experience. Summary of the Invention

[0004] This application provides a data processing method and related apparatus that can dynamically adjust the data format of business data in shared memory, thereby solving the problem of data format mismatch caused by dynamically adding business data and improving the user experience of the target object.

[0005] In a first aspect, embodiments of this application provide a data processing method. The data processing method includes: obtaining a first data description file, the first data description file being used to describe the data format of first business data, the first business data being business data obtained when the loading service program is in a first state; if the version of the first data description file is higher than the version of a second data description file, then determining data difference information between the first data description file and the second data description file, wherein the second data description file is used to describe the data format of second business data stored in shared memory, the second business data being business data obtained before the loading service program is in the first state, and the first business data and the second business data are different; and adjusting the data format of the second business data based on the data difference information and the data format of the first business data.

[0006] Secondly, embodiments of this application provide a data processing apparatus. This data processing apparatus includes an acquisition unit and a processing unit. The acquisition unit is used to acquire a first data description file, which describes the data format of first business data, which is business data obtained when the loading service program is in a first state. The processing unit is used to determine data difference information between the first and second data description files when the version of the first data description file is higher than the version of a second data description file, wherein the second data description file describes the data format of second business data stored in shared memory, and the second business data is business data obtained before the loading service program is in the first state, and the first and second business data are different; the processing unit is used to adjust the data format of the second business data based on the data difference information and the data format of the first business data.

[0007] In some optional implementations, the acquisition unit is used to acquire the file configuration information of the second data description file when the second data description file is stored in shared memory. The processing unit is used to acquire the first data description file from the configuration file when the file configuration information of the target data description file is different from the file configuration information of the second data description file, wherein the target data description file is the data description file of the currently running service program.

[0008] In some alternative embodiments, the acquisition unit is used to acquire the version number of the target data description file from the file configuration information of the target data description file, and to acquire the version number of the second data description file from the file configuration information of the second data description file. The processing unit is used to determine that the file configuration information of the target data description file and the file configuration information of the second data description file are different when the version number of the target data description file and the version number of the second data description file are different.

[0009] In some alternative implementations, the processing unit is further configured to traverse the message structure in the second data description file to obtain the fields in the message structure; obtain the field type of the current field in the data format of the second business data from the fields in the message structure; and adjust the field data corresponding to the field type of the current field according to the field length in the data format of the first business data based on the data difference information.

[0010] In some alternative implementations, the processing unit is used to adjust the field data corresponding to the basic type based on data difference information and according to the first field length when the field type of the current field is the basic type. Here, the first field length is the minimum value between the field length of the basic type in the data format of the first business data and the field length of the basic type in the second business data.

[0011] In some alternative implementations, the processing unit is used to adjust the field data corresponding to the string type according to the second field length based on data difference information when the field type of the current field is a string type. The second field length is the minimum value between the field length of the string type in the data format of the first business data and the field length of the string type corresponding to the second business data.

[0012] In some alternative implementations, the processing unit is configured to: when the field type of the current field includes an array type, traverse the subfields contained in the field corresponding to the array type to obtain the field type of the current subfield; and, based on data difference information, adjust the field data corresponding to the field type of the current subfield in the second business data according to the target field length. The target field length is obtained based on the field length of the array type in the data format of the first business data and the field length of the array type corresponding to the second business data.

[0013] In some alternative implementations, the processing unit is configured to adjust the field data corresponding to the basic type in the second business data according to the data difference information and a first target field length when the field type of the current sub-field is the basic type. The first target field length is the minimum value between the field length of the basic type in the data format of the first business data and the field length of the basic type in the second business data.

[0014] In some alternative implementations, the processing unit is configured to adjust the field data corresponding to the string type in the second business data according to the second target field length, based on data difference information, when the field type of the current subfield is a string type. The second target field length is the minimum value between the field length of the string type in the data format of the first business data and the field length of the string type corresponding to the string type in the second business data.

[0015] In some alternative embodiments, the acquisition unit is used to acquire file configuration information of a first data description file and file configuration information of a second data description file. The file configuration information of the first data description file includes version information and data information of the first data description file. The processing unit is used to determine data difference information between the first data description file and the second data description file based on the file configuration information of the first data description file and the second data description file.

[0016] A third aspect of this application provides a data processing apparatus, including: a memory, a transceiver, a processor, and a bus system. The bus system connects the memory and the processor to enable communication between them. The memory stores a program. The processor executes the program in the memory to perform the method corresponding to the first aspect and any possible implementation thereof.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method corresponding to the first aspect and any possible implementation thereof.

[0018] The fifth aspect of this application provides a computer program product containing instructions that, when run on a computer or processor, causes the computer or processor to execute the method described above for performing the first aspect and any possible implementation thereof.

[0019] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0020] In this embodiment, since the first business data is obtained when the loading service program is in the first state, the first data description file can describe the data format of the first business data. Furthermore, since the second business data is obtained before the loading service program is in the first state, the second data description file can also describe the data format of the second business data stored in shared memory. Moreover, the first business data and the second business data are different. Therefore, after obtaining the first data description file, by comparing the versions of the first and second data description files, and if the version of the first data description file is higher than the version of the second data description file, the data difference information between the first and second data description files can be determined. In this way, the data format of the second business data already stored in shared memory can be adjusted based on the data difference information and the data format of the first business data. Using the above method, when the version of the first data description file is higher than the version of the second data description file, it indicates that the data format of the first business data generated by the currently running service program has changed. At this time, the data format of the second business data in the old version of the second description file can be adjusted based on the new data format of the first business data and the data difference information between the description files. This eliminates the need to reserve additional shared memory space and also allows the data format of the second business data to be adjusted to be consistent with the data format of the first business data when adding or deleting certain business data based on the second business data. This solves the problem of data format mismatch caused by dynamically adding business data and improves the user experience of the target object. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This illustration shows an application scenario diagram of the data processing method provided in an embodiment of this application;

[0023] Figure 2A This illustration shows another application scenario of the data processing method provided in the embodiments of this application;

[0024] Figure 2B This illustration shows a schematic diagram of a blockchain structure provided in an embodiment of this application;

[0025] Figure 3 A flowchart of a data processing method provided in an embodiment of this application is shown;

[0026] Figure 4 This paper illustrates a flowchart of the adjustment process for different field types in the second data structure according to an embodiment of this application.

[0027] Figure 5 A schematic diagram of the data processing apparatus provided in an embodiment of this application is shown;

[0028] Figure 6 A schematic diagram of the hardware structure of the data processing apparatus provided in an embodiment of this application is shown. Detailed Implementation

[0029] This application provides a data processing method and related apparatus that can dynamically adjust the data format of business data in shared memory, thereby solving the problem of data format mismatch caused by dynamically adding business data and improving the user experience of the target object.

[0030] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that implementations of the application described herein can be implemented, for example, in sequences other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Figure 1 This illustration shows an application scenario diagram of a data processing method provided in an embodiment of this application. For example... Figure 1As shown, this application scenario may include shared memory 100 and database 200. The shared memory 100 and database 200 can transmit data via protocols such as Protobuf.

[0034] The described shared memory 100 refers to a large-capacity memory in a multiprocessor computer system that can be accessed by different central processing units (CPUs). It is a communication method between multiple processes, and multiple programs can also pass information through shared memory. In this embodiment, the data in the shared memory can be business data generated by the service program in different states. The shared memory 100 can be memory in a server, etc., and this embodiment does not provide specific limitations.

[0035] Database 200 can be a relational database, a non-relational database, or a key-value database, etc., and no specific limitation is made in this embodiment. Database 200 can store data in shared memory 100, and can also read data from database 200 and write it into shared memory 100.

[0036] In some examples, the data processing method provided in the embodiments of this application can also be applied to... Figure 2A The data sharing system shown refers to a system used for data sharing between nodes. This data sharing system may include multiple nodes 300, which can refer to various clients within the system. Each node 300, during normal operation, can receive input information and maintain shared data within the system based on this information. To ensure interoperability within the data sharing system, information connections can exist between each node, allowing for information transmission. For example, when any node in the data sharing system receives input information, other nodes in the system obtain this input information according to a consensus algorithm and store it as part of the shared data, ensuring consistency of data stored across all nodes.

[0037] Each node in the data sharing system has a corresponding node identifier, and each node can also store the node identifiers of other nodes in the data sharing system. This allows for the subsequent broadcasting of generated blocks to other nodes in the data sharing system based on their node identifiers. Each node can maintain a list of node identifiers, storing the node name and its corresponding node identifier in this list. The node identifier can be an internet protocol (IP) address or any other information that can be used to identify the node.

[0038] Each node in the data-sharing system stores the same blockchain. A blockchain consists of multiple blocks; see [link to blockchain documentation]. Figure 2B A blockchain consists of multiple blocks. The genesis block includes a block header and a block body. The block header stores input information feature values, version number, timestamp, and difficulty value, while the block body stores the input information. The next block after the genesis block takes the genesis block as its parent block. The next block also includes a block header and a block body. The block header stores the input information feature values ​​of the current block, the block header feature values ​​of the parent block, version number, timestamp, and difficulty value, and so on. This ensures that the block data stored in each block is related to the block data stored in the parent block, guaranteeing the security of the input information in the blocks.

[0039] It should be noted that the data processing method provided in this application embodiment can also be applied to devices such as servers. The server mentioned can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, a cloud server providing basic cloud computing services such as cloud computing, cloud services, cloud databases, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, or a server whose status needs frequent updates, such as a game server or a data caching server. This application does not specifically limit the application. Furthermore, the server mentioned can also be a server that directly interacts with the client. By using the data processing method provided in this application embodiment, high-frequency access, real-time read / write, online hot updates, and dynamic data scaling can be achieved. In addition, the described data processing method can also be applied to application scenarios with high-frequency object access and high real-time service response requirements, application scenarios where service programs need frequent updates and support for hot updates during online operation, and application scenarios where the size of completed data nodes is the same. Specifically, this application embodiment does not limit the application scenarios of the data processing method. In order to better understand the data processing method provided in the embodiments of this application, the following will describe the data processing method provided in the embodiments of this application in detail, taking the server as the execution subject.

[0040] Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application is shown. Figure 3 As shown, the data processing method may include the following steps:

[0041] 301. Obtain the first data description file. The first data description file is used to describe the data format of the first business data. The first business data is the business data obtained when the loading service program is in the first state.

[0042] In this example, when the service program starts, the server needs to load and obtain the corresponding data description file. This data description file describes the type and format of each business data item in each shared memory node, providing scalability support for business data loading. When loading the service program in its first state, the server can obtain the first business data. Since the data format of the first business data can be described using the corresponding first data description file, the server can obtain the first data description file.

[0043] It should be noted that a service program can be understood as a functional module capable of providing service support to multiple applications (APPs). For example, in high-concurrency application scenarios such as gaming, a service program can provide services such as data access and data synchronization between different virtual players; this application does not specify the specifics. Furthermore, the described first state may include, but is not limited to, the service program's restart, update, or normal operation states; this application does not specify the specifics. The mentioned first data description file can be used to describe the data format of the first business data.

[0044] In some examples, the method of obtaining the first data description file can be implemented as follows: when the second data description file is stored in shared memory, obtain the file configuration information of the second data description file; when the file configuration information of the target data description file is different from the file configuration information of the second data description file, obtain the first data description file from the configuration file.

[0045] In this example, the second data description file can be used to describe the data format of the second business data already stored in shared memory. The second business data is the business data obtained before the same service program was in its first state. Additionally, the target data description file can be understood as the data description file obtained when the service program is currently running. The file configuration information of the second data description file can be understood as the file content of the second data description file, such as, but not limited to, the version information and data information of the second data description file. The file configuration information corresponding to the target data description file can also be understood as the file content corresponding to the target data description file, such as including the version information and data information of the target data description file.

[0046] In this way, while the service program is running, the acquired target data description file can be written to the same shared memory. Then, the file configuration information of the target data description file and the second data description file are compared. If the file configuration information of the target data description file and the second data description file are different, it indicates that the data format of the first business data generated by the currently running service program has changed, and thus the data format of the old version of the second business data needs to be scaled up and adjusted. Therefore, when the file configuration information of the target data description file and the second data description file are different, the server needs to use the newly acquired target data description file as the first data description file from the configuration file, so that the first data description file can be used as the adjustment object for adjusting the data format of the second business data in the second data description. It should be noted that the configuration file is a configuration file automatically loaded when the service program starts, which can be generated when business data changes, such as deleting or adding some or all of the business data.

[0047] It should be understood that if the file configuration information of the target data description file is the same as that of the second data description file, it means that the data format of the second business data originally stored in the shared memory has not changed. In this case, there is no need to perform scaling adjustment on the data format of the second business data, that is, there is no need to execute the subsequent steps 302 to 303.

[0048] Furthermore, if the second data description file is not stored in the shared memory, it indicates that the service program is a new program that is starting for the first time. The server cannot obtain the second business data from the shared memory, so there is no need to perform subsequent scaling adjustments on the second business data. In this case, the server can directly obtain the latest first data description file from the configuration file and load the first data description file into the shared memory, without needing to execute subsequent steps 302 to 303.

[0049] In other examples, since the file configuration information includes version information and data information of the corresponding data description file, and the version information includes, but is not limited to, the version number, the comparison of whether the file configuration information of the target data description file is the same as that of the second data description file can be achieved as follows: The server can obtain the version number of the target data description file from its file configuration information, and obtain the version number of the second data description file from its file configuration information. Then, the server compares whether the version numbers of the target and second data description files are the same. If the version numbers of the target and second data description files are different, the server determines that the file configuration information of the target and second data description files is different.

[0050] It should be noted that the above only describes whether the file configuration information is the same from the perspective of whether the version number of the data description file is the same. In practical applications, other methods can also be used to determine whether the file configuration information is the same. The specific embodiments of this application do not make specific limitations.

[0051] 302. When the version of the first data description file is higher than the version of the second data description file, the data difference information between the first data description file and the second data description file is determined. The second data description file is used to describe the data format of the second business data stored in the shared memory. The second business data is the business data obtained before the loading service program is in the first state. The first business data and the second business data are different.

[0052] In this example, after obtaining the first data description file, the server can determine whether the version of the first data description file is higher than the version of the second data description file. For example, since the file configuration information of the first data description file includes its version information, and the version information includes its version number, similarly, since the file configuration information of the second data description file includes its version information, and the version information includes its version number, the server can compare the version numbers of the first and second data description files. If the version number of the first data description file is greater than the version number of the second data description file, it can be determined that the version of the first data description file is higher than the version of the second data description file. It should be noted that the fact that the version of the first data description file is higher than the version of the second data description file indicates that the current first data description file corresponds to the latest business data, while the second data description file originally stored in shared memory corresponds to an older version of the business data.

[0053] In this way, after comparing and finding that the version of the first data description file is higher than the version of the second data description file, the server can determine the differences in the file content, such as version information and data information, between the first and second data description files, that is, determine the data difference information between the first and second data description files.

[0054] For example, the server can determine the data difference information between the first data description file and the second data description file in the following manner: The server obtains the file configuration information of the first data description file and the file configuration information of the second data description file. The file configuration information of the first data description file includes version information and data information. Then, the server determines the data difference information between the first and second data description files based on the file configuration information of the first and second data description files.

[0055] It should be noted that in practical applications, data differences between data description files can be determined through methods other than file configuration information, which will not be specifically described in this application. Furthermore, the described file configuration information can also be understood in reference to the memory information in step 301 above, and will not be elaborated upon here.

[0056] 303. Based on the data difference information and the data format of the first business data, adjust the data format of the second business data.

[0057] In this example, the data format of the first business data may include, but is not limited to, the data type and field length of the first business data, and the data format of the second business data may include, but is not limited to, the data type and field length of the second business data. After determining the data difference information, the server can adjust the data format of the second business data described in the second data description file according to the data format of the first business data described in the first data description file, so that the adjusted data format of the second business data is consistent with the data format of the first business data.

[0058] In some optional examples, the adjustment of the data format of the second business data in step 303 above can be implemented as follows: The server iterates through the message structure in the second data description file to obtain the fields in the message structure, and from the fields in the message structure, obtains the field type of the current field in the data format of the second business data. Then, based on the data difference information, the server adjusts the field data corresponding to the field type of the current field according to the field length in the data format of the first business data.

[0059] In this example, the second data description file may include, but is not limited to, a protobuf description file. The embodiments of this application will be described using only the protobuf description file as an example.

[0060] For example, taking a protobuf description file as the second data description file, the protobuf description file is the definition file for protobuf messages (PB messages for short), which defines the message structure. It should be noted that a single protobuf description file can define multiple message structures and can also reference messages defined in other protobuf description files. For each message structure, the protobuf description file provides descriptive information for at least one field constituting that message structure. The descriptive information for each field may include, but is not limited to, field type, field name, data type, etc., and this application does not impose specific limitations on this.

[0061] The field types can include: basic types, string types, array types, and composite types. Basic types may include, but are not limited to, bool, int8, uint8, int16, uint16, int32, uint32, int64, uint64, float, double, etc., without further limitation in this application. String types may include, but are not limited to, string arrays in C and / or C++, without further limitation in this application. Array types can also be understood as repeated types, indicating that the corresponding field can be repeated any number of times, and the order of repeated data will be stored in the protobuf buffer. Composite types refer to nested messages; the field types of the nested message are composite types, including but not limited to nested basic types, string types, and array types, without further limitation in this application. Data types are consistent with the data types of various programming languages, including but not limited to int32, string, float, etc. Field names are basically the same as variable naming conventions in languages ​​such as C, C++, and Java, for example, page, num, etc.

[0062] Additionally, the array type corresponds to arrays in C and C++, which can store data types corresponding to basic types, string types, and composite types. A composite type refers to a message nested within another message, where the nested message can contain basic types, string types, and array types. Different adjustment methods can be applied to different field types in the message structure of the second data description file; details can be found in [reference needed]. Figure 4 The following will be explained using the flowcharts showing the adjustment processes for different field types. Figure 4 This will be explained from the perspectives of basic types, string types, array types, and composite types. Please refer to the following descriptions for a detailed understanding:

[0063] (I) Field type is a basic type

[0064] In some examples, when the current field's field type is a base type, the server can adjust the field data corresponding to the base type based on the data difference information and according to the length of the first field.

[0065] In this example, the first field length described is the minimum of the field lengths of the basic type in the data format of the first business data and the field lengths of the basic type in the second business data. For example, if the field length of the basic type in the second business data is 128 bytes, and the actual storage size of each memory node in the shared memory is 64 bytes, then the field length of the basic type in the first business data can be set to 64 bytes. At this point, the server can adjust the field length of the basic type in the second business data (e.g., 128 bytes) to multiple fields with a length of 64 bytes based on the data difference information and the field length of the basic type in the data format of the first business data (e.g., 64 bytes). This ensures that the data format of the adjusted second business data is consistent with that of the first business data, and the cache space for the business data can be dynamically adjusted without reserving additional cache space in the shared memory. The server can also copy the original 128 bytes of the second business data to the shared memory according to the newly adjusted field lengths.

[0066] Similarly, if the basic type field length in the second business data is 32 bytes, and the actual storage size of each memory node in the shared memory is 64 bytes, then the basic type field length in the first business data can be set to 64 bytes. In this case, the server can adjust the basic type field length in the second business data (e.g., 32 bytes) to 64 bytes based on the data difference information and the basic type field length in the first business data's data format (e.g., 64 bytes). This ensures that the adjusted second business data's data format is consistent with the first business data's data format, and the cache space for the business data can be dynamically adjusted without reserving additional cache space in the shared memory. The server can then copy the original 32-byte second business data to the shared memory according to the newly adjusted field length.

[0067] (ii) The field type is string.

[0068] In some examples, when the current field type is string, the field data corresponding to the string type is adjusted according to the length of the second field based on the data difference information.

[0069] In this example, the length of the second field is the minimum of the length of the string type field in the data format of the first business data and the length of the corresponding string type field in the second business data. For example, if the length of the string type field in the second business data is 128 bytes, and the actual storage size of each memory node in the shared memory is 64 bytes, then the length of the string type field in the first business data can be set to 64 bytes. At this point, the server can adjust the length of the string type field in the second business data (e.g., 128 bytes) to multiple fields with a length of 64 bytes based on the data difference information and the length of the string type field in the data format of the first business data (e.g., 64 bytes). This ensures that the data format of the adjusted second business data is consistent with that of the first business data, and the cache space for the business data can be dynamically adjusted without reserving additional cache space in the shared memory. The server can also copy the original 128 bytes of the second business data to the shared memory according to the newly adjusted field lengths.

[0070] Similarly, if the string field length in the second business data is 32 bytes, and the actual storage size of each memory node in the shared memory is 64 bytes, then the string field length in the first business data can be set to 64 bytes. In this case, the server can adjust the string field length in the second business data (e.g., 64 bytes) to 64 bytes based on the data difference information and the string field length in the first business data (e.g., 64 bytes). This ensures that the adjusted second business data format is consistent with the first business data format, and the cache space for the business data can be dynamically adjusted without reserving additional cache space in the shared memory. The server can then copy the original 32-byte second business data to the shared memory according to the newly adjusted field length.

[0071] (iii) Field type is array type

[0072] In some examples, since the array type is a repeated type, the fields corresponding to the array type can be divided into different types of subfields, such as the basic type, string type, composite type, etc. as described above. This application embodiment only uses the division of array type into basic type, string type, composite type, etc. as an example for illustration.

[0073] For example, when the field type is an array, the server can iterate through the subfields contained in the field corresponding to the array type to obtain the field type of the current subfield. Then, based on the data difference information, the server adjusts the field data corresponding to the field type of the current subfield in the second business data according to the target field length. The target field length is obtained based on the field length of the array type in the data format of the first business data and the field length of the array type corresponding to the second business data.

[0074] Different adjustment methods can also be applied to each subfield of this array type. Please refer to the following explanation for details:

[0075] ① The field type of the subfield is the basic type.

[0076] In some examples, when the field type of a subfield in an array type is a base type, the server can adjust the field data corresponding to the base type in the second business data according to the length of the first target field based on the data difference information, when the field type of the current subfield is a base type.

[0077] It should be noted that the first target field length is the minimum of the field lengths of the basic types in the data format of the first business data and the field lengths of the basic types in the second business data. Furthermore, in this embodiment, the server adjusts the field data corresponding to the basic types in the second business data according to the first target field length based on the data difference information. For details, please refer to the basic types described in section (I) above; further explanation is not provided here.

[0078] ② The field type of the subfield is string.

[0079] In some examples, when the field type of a subfield in an array type is string, the server can adjust the field data corresponding to the string type in the second business data according to the length of the second target field based on the data difference information.

[0080] It should be noted that the length of the second target field is the minimum of the length of the string type field in the data format of the first business data and the length of the corresponding string type field in the second business data. Furthermore, in this embodiment, the server adjusts the string type field data in the second business data according to the second target field length based on the data difference information. For details, please refer to the string type content described in section (II) above; further explanation is not provided here.

[0081] (iv) The field type is a composite type

[0082] In this example, since a composite type refers to a message nested within another message, and that other message can contain basic types, string types, and array types, then in the combination... Figure 4 The flowchart shown illustrates that each subtype within a composite type ultimately recursively returns to the basic type and the string type. Therefore, the adjustment process for composite types can be understood by referring to the content in sections (I) to (III) above, and will not be elaborated upon here.

[0083] Thus, in accordance with Figure 4 Once the adjustment process has completed the adjustment of the data format of the second business data, the second business data can be copied to the shared node according to the new data format.

[0084] For example, based on the above description, the following is an illustrative example of a protobuf description file in an embodiment of this application. For example:

[0085]

[0086]

[0087] It should be understood that in the protobuf description file shown above, PBMessage1 is the message structure included in the PB Message of the second business data in the second data description file, specifically including the following four data types: PB Pod type, PBString type, PB Repeated type, and PB Message type. Specifically, 1. The PBPod1 field corresponds to the PB Pod type, which can be an int32 type in C language. 2. The PBString field corresponds to the PB String type, which can be a string type in C language with a length of 64 char variables. 3. The PBRepeated field corresponds to the PB Repeated type, which can be an array type in C language, with the array size defined as 100. 4. The PBMessage2 field corresponds to the PBMessage type, which can be the custom type PBInnerMesage, a struct type in C language. The described PBInnerMesage is the type defined in PBMessage1, containing one data type, namely the PB Pob type. The described PBPod2 field corresponds to the PB Pob type, which can be an int32 type in C language.

[0088] It should be understood that the PB Pod type, PB String type, PB Repeated type, and PBMessage type described above can correspond to the basic type, string type, array type, and composite type described in step 303 above.

[0089] It is understood that this example is only used to introduce the adjustment of field length for different field types. In practical applications, the protobuf description file can be other instances, and this application embodiment does not impose any specific limitations.

[0090] In this embodiment, since the first business data is obtained when the loading service program is in the first state, the first data description file can describe the data format of the first business data. Furthermore, since the second business data is obtained before the loading service program is in the first state, the second data description file can also describe the data format of the second business data stored in shared memory. Moreover, the first business data and the second business data are different. Therefore, after obtaining the first data description file, by comparing the versions of the first and second data description files, and if the version of the first data description file is higher than the version of the second data description file, the data difference information between the first and second data description files can be determined. In this way, the data format of the second business data already stored in shared memory can be adjusted based on the data difference information and the data format of the first business data. By employing the above method, when the version of the first data description file is higher than the version of the second data description file, it indicates that the data format of the first business data generated by the currently running service program has changed. In this case, the data format of the second business data in the old version of the second description file can be adjusted based on the new data format of the first business data and the data difference information between the description files. This eliminates the need to reserve additional shared memory space and ensures that when adding or deleting business data based on the second business data, the data format of the second business data is adjusted to match that of the first business data. This solves the problem of data format mismatch caused by dynamically adding business data, improving the user experience for the target object. Furthermore, the adjusted second business data can also be stored in shared memory, so that after the service program restarts, updates, or abnormally exits and resumes, it is not necessary to reload the business data from the database. This allows for more persistent storage of business data, and the object is unaware of service program updates, improving the user experience for the object and other products.

[0091] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that to achieve the above functions, corresponding hardware structures and / or software modules are included to execute each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0092] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0093] The data processing apparatus in the embodiments of this application will now be described in detail. Figure 5 This is a schematic diagram of one embodiment of the data processing apparatus provided in this application. Figure 5 As shown, the data processing apparatus may include an acquisition unit 501 and a processing unit 502. The acquisition unit 501 is used to acquire a first data description file, which describes the data format of first business data, which is business data obtained when the loading service program is in a first state. The processing unit 502 is used to determine data difference information between the first and second data description files when the version of the first data description file is higher than the version of the second data description file, wherein the second data description file describes the data format of second business data stored in shared memory, and the second business data is business data obtained before the loading service program is in the first state, and the first and second business data are different; the processing unit 502 is used to adjust the data format of the second business data based on the data difference information and the data format of the first business data.

[0094] In some optional implementations, the acquisition unit 501 is used to acquire the file configuration information of the second data description file when the second data description file is stored in shared memory. The processing unit 502 is used to acquire the first data description file from the configuration file when the file configuration information of the target data description file is different from the file configuration information of the second data description file, wherein the target data description file is the data description file of the currently running service program.

[0095] In some alternative embodiments, the acquisition unit 501 is used to acquire the version number of the target data description file from the file configuration information of the target data description file, and to acquire the version number of the second data description file from the file configuration information of the second data description file. The processing unit 502 is used to determine that the file configuration information of the target data description file is different from the file configuration information of the second data description file when the version number of the target data description file is different from the version number of the second data description file.

[0096] In some alternative implementations, the processing unit 502 is further configured to traverse the message structure in the second data description file to obtain the fields in the message structure; obtain the field type of the current field in the data format of the second business data from the fields in the message structure; and adjust the field data corresponding to the field type of the current field according to the field length in the data format of the first business data based on the data difference information.

[0097] In some alternative embodiments, the processing unit 502 is used to adjust the field data corresponding to the basic type according to the data difference information and the first field length when the field type of the current field is the basic type. The first field length is the minimum value between the field length of the basic type in the data format of the first business data and the field length of the basic type in the second business data.

[0098] In some alternative implementations, the processing unit 502 is used to adjust the field data corresponding to the string type according to the second field length based on data difference information when the field type of the current field is a string type. The second field length is the minimum value between the field length of the string type in the data format of the first business data and the field length of the string type corresponding to the second business data.

[0099] In some alternative implementations, the processing unit 502 is configured to: when the field type of the current field includes an array type, traverse the subfields contained in the field corresponding to the array type to obtain the field type of the current subfield; and, based on data difference information, adjust the field data corresponding to the field type of the current subfield in the second business data according to the target field length. The target field length is obtained based on the field length of the array type in the data format of the first business data and the field length of the array type corresponding to the second business data.

[0100] In some alternative embodiments, the processing unit 502 is used to adjust the field data corresponding to the basic type in the second business data according to the data difference information and a first target field length when the field type of the current sub-field is the basic type. The first target field length is the minimum value between the field length of the basic type in the data format of the first business data and the field length of the basic type in the second business data.

[0101] In some alternative implementations, the processing unit 502 is configured to adjust the field data corresponding to the string type in the second business data according to the second target field length, based on data difference information, when the field type of the current subfield is a string type. The second target field length is the minimum value between the field length of the string type in the data format of the first business data and the field length of the string type corresponding to the string type in the second business data.

[0102] In some alternative embodiments, the acquisition unit 501 is used to acquire file configuration information of a first data description file and file configuration information of a second data description file. The file configuration information of the first data description file includes version information and data information of the first data description file. The processing unit 502 is used to determine data difference information between the first data description file and the second data description file based on the file configuration information of the first data description file and the second data description file.

[0103] The data processing apparatus in the embodiments of this application has been described above from the perspective of modular functional entities. The data processing apparatus in the embodiments of this application will now be described below from the perspective of hardware processing. Figure 6 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. The data processing apparatus may include the server described above. The data processing apparatus may vary considerably depending on its configuration or performance. The data processing apparatus may include at least one processor 601, a communication line 607, a memory 603, and at least one communication interface 604.

[0104] The processor 601 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (server IC), or one or more integrated circuits used to control the execution of the program of the present application.

[0105] Communication line 607 may include a path for transmitting information between the aforementioned components.

[0106] Communication interface 604 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0107] The memory 603 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions. The memory can exist independently and be connected to the processor via communication line 607. The memory can also be integrated with the processor.

[0108] The memory 603 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 601. The processor 601 executes the computer execution instructions stored in the memory 603, thereby implementing the data processing method provided in the above embodiments of this application.

[0109] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0110] In a specific implementation, as one example, the data processing device may include multiple processors, such as... Figure 6 Processors 601 and 602 are described herein. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0111] In a specific implementation, as one embodiment, the data processing apparatus may further include an output device 605 and an input device 606. The output device 605 communicates with the processor 601 and can display information in various ways. The input device 606 communicates with the processor 601 and can receive user input in various ways. For example, the input device 606 may be a mouse, a touch screen device, or a sensing device, etc.

[0112] The aforementioned data processing device can be a general-purpose device or a special-purpose device. In specific implementations, the data processing device can be a microscope or something similar. Figure 6 A device with a similar structure. The embodiments of this application do not limit the type of this data processing device.

[0113] It should be noted that Figure 6 The processor 601 can invoke computer execution instructions stored in the memory 603 to cause the data processing device to perform actions such as... Figures 3 to 4 The method in the corresponding method embodiment.

[0114] Specifically, Figure 5 The function / implementation process of the processing unit 502 can be achieved through... Figure 6 The processor 601 in the memory calls computer execution instructions stored in the memory 603 to implement the function. Figure 5 The function / implementation process of the acquisition unit 501 can be achieved through... Figure 6 It is implemented using the communication interface 604.

[0115] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0116] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0117] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0120] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, they can be implemented in whole or in part in the form of a computer program product.

[0122] A computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, they generate, in whole or in part, the processes or functions according to embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., SSDs), etc.

[0123] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Obtain a first data description file, which is used to describe the data format of the first business data, which is the business data obtained when the loading service program is in the first state; When the version of the first data description file is higher than the version of the second data description file, the data difference information between the first data description file and the second data description file is determined. The second data description file is used to describe the data format of the second business data stored in the shared memory. The second business data is the business data obtained before the service program was loaded into the first state. The first business data is different from the second business data. Traverse the message structure in the second data description file to obtain the fields in the message structure; From the fields in the message structure, obtain the field type of the current field in the data format of the second business data; Based on the data difference information, the field data corresponding to the field type of the current field is adjusted according to the field length in the data format of the first business data, including: When the field type of the current field includes an array type, iterate through the subfields contained in the field corresponding to the array type and obtain the field type of the current subfield; When the field type of the current sub-field is a basic type, based on the data difference information, the field data corresponding to the basic type in the second business data is adjusted according to a first target field length, wherein the first target field length is the minimum value between the field length of the basic type in the data format of the first business data and the field length of the basic type in the second business data; or... When the field type of the current subfield is string, based on the data difference information, the field data corresponding to the string type in the second business data is adjusted according to the second target field length, wherein the second target field length is the minimum value between the field length of the string type in the data format of the first business data and the field length corresponding to the string type in the second business data.

2. The method according to claim 1, characterized in that, The acquisition of the first data description file includes: When the second data description file is stored in the shared memory, obtain the file configuration information of the second data description file; When the file configuration information of the target data description file is different from the file configuration information of the second data description file, the first data description file is obtained from the configuration file, wherein the target data description file is the data description file when the service program is currently running.

3. The method according to claim 2, characterized in that, The file configuration information of the target data description file is different from that of the second data description file, including: Obtain the version number of the target data description file from the file configuration information of the target data description file; Obtain the version number of the second data description file from the file configuration information of the second data description file; If the version number of the target data description file is different from the version number of the second data description file, then it is determined that the file configuration information of the target data description file is different from the file configuration information of the second data description file.

4. The method according to claim 1, characterized in that, The step of adjusting the field data corresponding to the field type of the current field based on the data difference information and according to the field length in the data format of the first business data includes: When the field type of the current field is a basic type, the field data corresponding to the basic type is adjusted according to the data difference information and the first field length, wherein the first field length is the minimum value between the field length of the basic type in the data format of the first business data and the field length of the basic type in the second business data.

5. The method according to claim 1, characterized in that, The step of adjusting the field data corresponding to the field type of the current field based on the data difference information and according to the field length in the data format of the first business data includes: When the field type of the current field is string, the field data corresponding to the string type is adjusted according to the data difference information and the second field length, wherein the second field length is the minimum value between the field length of the string type in the data format of the first business data and the field length of the string type corresponding to the second business data.

6. The method according to any one of claims 1 to 3, characterized in that, Determining the data difference information between the first data description file and the second data description file includes: Obtain the file configuration information of the first data description file and the file configuration information of the second data description file. The file configuration information of the first data description file includes the version information and data information of the first data description file. Based on the file configuration information of the first data description file and the file configuration information of the second data description file, the data difference information between the first data description file and the second data description file is determined.

7. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire a first data description file, which describes the data format of the first business data, and the first business data is the business data obtained when the loading service program is in the first state. The processing unit is configured to determine data difference information between the first data description file and the second data description file when the version of the first data description file is higher than the version of the second data description file, wherein the second data description file is used to describe the data format of the second business data stored in shared memory, the second business data is business data obtained before the service program is loaded into the first state, and the first business data is different from the second business data. The processing unit is configured to traverse the message structure in the second data description file to obtain the fields in the message structure; and obtain the field type of the current field in the data format of the second business data from the fields in the message structure. The processing unit is configured to adjust the field data corresponding to the field type of the current field according to the field length in the data format of the first business data based on the data difference information, including: When the field type of the current field includes an array type, iterate through the subfields contained in the field corresponding to the array type and obtain the field type of the current subfield; When the field type of the current sub-field is a basic type, based on the data difference information, the field data corresponding to the basic type in the second business data is adjusted according to a first target field length, wherein the first target field length is the minimum value between the field length of the basic type in the data format of the first business data and the field length of the basic type in the second business data; or... When the field type of the current subfield is string, based on the data difference information, the field data corresponding to the string type in the second business data is adjusted according to the second target field length, wherein the second target field length is the minimum value between the field length of the string type in the data format of the first business data and the field length corresponding to the string type in the second business data.

8. The data processing apparatus according to claim 7, characterized in that, The acquisition unit is used to acquire the file configuration information of the second data description file when the second data description file is stored in the shared memory; The processing unit is configured to obtain the first data description file from the configuration file when the file configuration information of the target data description file is different from the file configuration information of the second data description file, wherein the target data description file is the data description file currently running the service program.

9. The data processing apparatus according to claim 8, characterized in that, The acquisition unit is configured to acquire the version number of the target data description file from the file configuration information of the target data description file; and acquire the version number of the second data description file from the file configuration information of the second data description file; The processing unit is configured to determine that the file configuration information of the target data description file is different from the file configuration information of the second data description file when the version number of the target data description file is different from the version number of the second data description file.

10. The data processing apparatus according to claim 7, characterized in that, The processing unit is used to adjust the field data corresponding to the basic type based on the data difference information when the field type of the current field is the basic type, according to the first field length, wherein the first field length is the minimum value between the field length of the basic type in the data format of the first business data and the field length of the basic type in the second business data.

11. The data processing apparatus according to claim 7, characterized in that, The processing unit is used to adjust the field data corresponding to the string type according to the data difference information and the second field length when the field type of the current field is a string type. The second field length is the minimum value between the field length of the string type in the data format of the first business data and the field length of the string type corresponding to the string type in the second business data.

12. The data processing apparatus according to any one of claims 7 to 9, characterized in that, The acquisition unit is used to acquire the file configuration information of the first data description file and the file configuration information of the second data description file. The file configuration information of the first data description file includes the version information and data information of the first data description file. The processing unit is used to determine the data difference information between the first data description file and the second data description file based on the file configuration information of the first data description file and the file configuration information of the second data description file.

13. A data processing apparatus, characterized in that, The data processing device includes: an input / output (I / O) interface, a processor, and a memory, wherein the memory stores program instructions; The processor is configured to execute the program instructions stored in the memory to perform the method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 6.

15. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 6.

Images