Field processing method, device, equipment, computer-readable storage medium and product

Abstract

The present disclosure provides a field processing method, apparatus, device, computer-readable storage medium and product, which relate to the field of data processing, and in particular to the field of big data. The specific implementation scheme is as follows: obtaining the to-be-processed fields corresponding to the data requests associated with multiple preset external services; determining multiple target fields in the to-be-processed fields that match the preset field list, wherein the preset field list includes the first unused field associated with each external service, and the preset field list is obtained by deleting the first associated field of the external service collected based on the target collection parameters based on the preset use field list; performing a deletion operation on the target field to obtain the remaining fields; and performing data processing operations based on the remaining fields. In this way, the field offline operation can be guided based on the preset field list to avoid system performance problems caused by the full field collection.

Description

Technical Field

[0001] The present disclosure relates to big data in data processing, and more particularly to a field processing method, apparatus, device, computer-readable storage medium, and product. Background Art

[0002] In modern backend service architectures, systems often need to obtain data from multiple external services or APIs. This data is stored in the form of structure fields to support business logic and application functions.

[0003] As product and business evolve, the number of fields requested and stored by the system continues to increase. This expansion of fields can lead to decreased system performance, and excessive fields can increase learning costs, reduce iteration efficiency, and even affect system stability. Summary of the Invention

[0004] The present disclosure provides a field processing method, apparatus, device, computer-readable storage medium, and product for guiding field offline operations to avoid system performance issues caused by full field collection.

[0005] According to a first aspect of the present disclosure, a field processing method is provided, comprising:

[0006] Get the pending fields corresponding to the data requests associated with multiple preset external services;

[0007] Determining a plurality of target fields among the to-be-processed fields that match a preset field list, wherein the preset field list includes a first unused field associated with each of the external services, the preset field list being obtained by deleting the first associated fields of the external services collected based on the target collection parameters based on a preset used field list;

[0008] Perform a deletion operation on the target field to obtain the remaining fields;

[0009] A data processing operation is performed based on the remaining fields.

[0010] According to a second aspect of the present disclosure, a field processing device is provided, including:

[0011] An acquisition module is used to obtain the to-be-processed fields corresponding to the data requests associated with multiple preset external services;

[0012] a determination module, configured to determine a plurality of target fields among the fields to be processed that match a preset field list, wherein the preset field list includes a first unused field associated with each of the external services, the preset field list being obtained by deleting the first associated fields of the external services collected based on the target collection parameters based on a preset used field list;

[0013] A deletion module, configured to perform a deletion operation on the target field to obtain the remaining fields;

[0014] A processing module is used to perform data processing operations based on the remaining fields.

[0015] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0021] The technology disclosed herein solves the problem of system performance degradation caused by full data collection for multiple external services.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0024] Figure 1 This is a diagram of the system architecture on which the present disclosure is based;

[0025] Figure 2 A flowchart of a field processing method provided in an embodiment of the present disclosure;

[0026] Figure 3 A flowchart of a field processing method provided in yet another embodiment of the present disclosure;

[0027] Figure 4 A flowchart of a field processing method provided in yet another embodiment of the present disclosure;

[0028] Figure 5 A flowchart of a field processing method provided in yet another embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of the structure of a field processing device provided in an embodiment of the present disclosure;

[0030] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] The present disclosure provides a field processing method, apparatus, device, computer-readable storage medium, and product, which are applied to big data in the field of data processing to identify and guide offline useless fields, achieve efficient field management, and optimize system performance.

[0033] In modern backend service architectures, systems often need to communicate with multiple external services to obtain data for subsequent business processing. For example, in order to accurately recommend content, recommendation systems need to obtain data such as forward ranking data, reverse ranking data, resource granularity data, and historical data from multiple external services.

[0034] As products and services evolve, the number of fields requested and stored by the system continues to increase. This expansion of fields can lead to decreased system performance, and excessive fields can increase learning costs, reduce iteration efficiency, and even affect system stability.

[0035] Related technologies typically collect all data requested by external services. However, with the continuous iteration of product services, the number of fields requested and stored by the system continues to increase. Full data collection often leads to reduced system performance, and excessive fields increase learning costs, reduce iteration efficiency, and even affect system stability.

[0036] In the process of solving the above technical problems, the inventors discovered through research that there is a lot of data associated with external services, among which there are often some fields that are not used by external services. Therefore, in order to accurately obtain available data, the fields in the service can be collected in advance, and a preset field list consisting of all unused data can be constructed. Therefore, in the subsequent data acquisition process, the preset field list can be used to guide the offline of useless fields. In this way, only the data that needs to be used can be acquired. This effectively reduces the amount of data that needs to be acquired and stored, and realizes efficient field management and optimized system performance. It can not only reduce the waste of system resources, but also improve development and maintenance efficiency, ensuring that the system remains stable and efficient in a rapidly changing business environment.

[0037] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0038] In order to enable readers to have a deeper understanding of the implementation principle of this disclosure, the following Figure 1-Figure 7 The embodiments of the present disclosure are further refined.

[0039] Figure 1 This is the system architecture diagram on which the present disclosure is based, such as Figure 1 As shown, the system architecture based on which the present disclosure is based includes at least a server 11 and a plurality of external servers 12. The server 11 is respectively connected to each external server for communication.

[0040] Based on the above system architecture, after obtaining the pending fields corresponding to all data requests associated with the external server 12, the server 11 can offline unused fields in the pending fields based on a pre-built preset field list, thereby achieving efficient field management and optimizing system performance.

[0041] Figure 2 A flow chart of a field processing method provided in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the method includes:

[0042] Step 201: Obtain fields to be processed corresponding to data requests associated with multiple preset external services.

[0043] The subject matter of this embodiment is a field processing device. The field processing device may be coupled to a server. The server may be a server associated with a preset system. For example, it may be a server associated with a recommendation system. The server may be connected to multiple external services.

[0044] In this embodiment, the server can obtain the to-be-processed fields corresponding to all data requests associated with multiple preset external services, so that data processing operations can be performed based on the to-be-processed fields.

[0045] For example, the field processing device can be coupled to a server associated with the recommendation system. To implement content recommendations, the recommendation system needs to obtain data such as forward ranking data, reverse ranking data, resource granularity data, and historical data from multiple external services. This data can then be processed to accurately implement content recommendations.

[0046] As the business continues to evolve, the number of fields requested and stored by the recommendation system continues to increase. This expansion can lead to decreased system performance, and excessive fields can increase learning costs, reduce iteration efficiency, and even affect system stability. Therefore, after obtaining the pending fields corresponding to all data requests associated with multiple preset external services, you can also delete fields that are not used by external services.

[0047] Step 202: Determine multiple target fields in the fields to be processed that match a preset field list, wherein the preset field list includes the first unused field associated with each external service, and the preset field list is obtained by deleting the first associated field of the external service collected based on the target collection parameters based on the preset used field list.

[0048] In this embodiment, in order to implement offline guidance operations for fields not used by external services, a preset field list may be pre-built, wherein the preset field list pre-stores the first unused field associated with each external service.

[0049] Optionally, a used field list can be pre-built based on all fields used by multiple external services during data requests. Therefore, after obtaining the first associated field corresponding to the full request associated with multiple external services, the used fields in the first associated field can be deleted based on the used field list. A preset field list is then constructed based on the remaining fields after deletion.

[0050] Therefore, after obtaining the pending fields corresponding to all data requests associated with multiple preset external services, multiple target fields in the pending fields that match the preset field list are determined, and the preset field list pre-stores all unused fields associated with each external service.

[0051] Step 203: Perform a deletion operation on the target field to obtain remaining fields.

[0052] Step 204: Perform data processing operations based on the remaining fields.

[0053] In this embodiment, after determining multiple target fields in the pending fields that are not used by the external service, in order to avoid the problem of excessive storage of fields causing system performance degradation, the multiple target fields in the total pending fields can be deleted. Therefore, after the deletion, the remaining fields in the pending fields are the fields used by the external service, and therefore data processing operations can be performed based on the remaining fields in the total pending fields.

[0054] The field processing method provided in this embodiment pre-builds a preset field list that includes all unused fields in external services. After obtaining fields to be processed from multiple external services, this preset field list can be used to guide the offline operation of unused fields in the fields to be processed. Consequently, content storage and data processing operations can be performed only on the used fields in the fields to be processed. This enables efficient management of fields to be processed and improves data processing efficiency.

[0055] Furthermore, based on any of the above embodiments, the method further includes:

[0056] After constructing the used field list, at least one round of field collection operation is performed on the data request to obtain a first associated field, wherein the used field list is constructed based on the initial fields collected in the first round of field collection operation.

[0057] The preset field list is determined based on the first associated field and the used field list.

[0058] In this embodiment, in order to implement offline guidance operations for fields not used by external services, a preset field list may be pre-built, wherein the preset field list pre-stores all unused fields associated with each external service.

[0059] In order not to affect the system service, the preset field list can be constructed during the operation of the system service.

[0060] Optionally, multiple rounds of field collection operations may be performed on all data requests associated with multiple external requests, and a preset field list may be determined based on the first associated fields collected through the multiple rounds of field collection operations and the used field list.

[0061] First, a first round of field collection can be performed according to the initial collection parameters. A used field list is constructed based on the initial fields collected in the first round of field collection. Therefore, after constructing the used field list, at least one round of field collection can be performed on the data request to obtain the first associated fields. By pre-constructing the used field list, in each subsequent round of field collection, the used fields in the collected second associated fields can be deleted based on the used field list to obtain unused fields.

[0062] It should be noted that in order to avoid the impact of the preset field list construction operation on system services, during the initial collection process, the field collection operation can be performed according to a smaller collection flow and collection duration. For example, all fields can be collected according to a collection flow of 1% of the full data request, and the collection duration can be set to one hour. During each round of field collection operation, the collection parameters are continuously adjusted and expanded to cover more field usage scenarios. Until the collection flow reaches 100% and the collection duration reaches a full day, the field collection operation is completed, and a preset field list is constructed based on the unused fields in the first associated field currently collected.

[0063] The field processing method provided in this embodiment pre-processes multiple rounds of field collection for all data requests associated with multiple external requests. This allows the preset field list to be determined based on the collected first associated fields and the list of used fields. This allows subsequent offline operations of fields to be accurately guided based on this preset field list, improving the accuracy of field management and the efficiency of field-based data processing.

[0064] Figure 3 A flow chart of a field processing method provided in another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 3 As shown, determining the preset field list based on the first associated field and the used field list includes:

[0065] Step 301: Determine the historical collection parameters corresponding to the previous round of field collection operations.

[0066] Step 302: Determine the second associated field collected in the previous round of field collection operation.

[0067] Step 303: Determine and adjust acquisition parameters based on the historical acquisition parameters and the second associated field.

[0068] Step 304: Execute the next round of field collection operation according to the adjusted collection parameters.

[0069] Step 305: In response to determining that the adjusted acquisition parameter meets a preset termination condition, the adjusted acquisition parameter is determined as the target acquisition parameter, and the field acquired in the next round of field acquisition operation performed based on the target acquisition parameter is determined as the first associated field.

[0070] Step 306: Delete the fields in the first associated fields that match the used fields in the used field list to obtain the preset field list.

[0071] In this embodiment, multiple rounds of field collection operations can be performed on all data requests associated with multiple external requests. During each round of field collection operations, collection parameters are continuously adjusted and expanded to cover more field usage scenarios.

[0072] Optionally, to cover more field usage scenarios, adjusted collection parameters for this round can be determined based on the historical collection parameters from the previous round and the percentage of unused fields in the second associated fields collected in the previous round. Therefore, the historical collection parameters corresponding to the previous round of field collection operations can be determined, as well as the second associated fields collected in the previous round of field collection operations. Adjusted collection parameters are determined based on the historical collection parameters and the second associated fields. The next round of field collection operations is performed on the second associated fields of multiple external services according to the adjusted collection parameters.

[0073] It should be noted that before constructing the preset field list, it is necessary to first construct the used field list. Among them, the initial field can be collected according to the preset initial parameters, and the full amount of used fields can be identified in the initial field to construct the used field list. Therefore, the collection operation of the initial field is the first round of field collection operation. The multiple rounds of field collection operations when constructing the preset field list are all based on the previous round of field collection operations.

[0074] Furthermore, in order to accurately construct the preset field list, a termination condition may be pre-set. After determining the adjusted acquisition parameters for this round and completing the field acquisition operation based on the adjusted acquisition parameters, it may be determined whether the adjusted acquisition parameters meet the termination condition.

[0075] In response to the adjusted collection parameters satisfying a preset termination condition, i.e., the current field collection operation has ended, the current adjusted collection parameters can be determined as the target collection parameters, and the fields collected by the field collection operation performed based on the target collection parameters can be determined as first associated fields. A preset field list is constructed based on the first associated fields. Fields in the first associated fields that match used fields in the preset used field list can be deleted to obtain the preset field list.

[0076] In response to the adjustment of the acquisition parameters not meeting the preset termination conditions, it indicates that the field acquisition operation has not ended and the next round of field acquisition operations needs to be further performed. At this time, the current field acquisition operation can be determined as the previous round of field acquisition operations, that is, the current adjustment acquisition parameters are determined as historical acquisition parameters, and the second associated fields collected in this round are determined as the second associated fields collected in the previous round. Return to the step of determining the historical acquisition parameters corresponding to the previous round of field acquisition operations. After multiple cycles of acquisition, when it is determined that the current adjustment acquisition parameters meet the preset termination conditions, the current adjustment acquisition parameters are determined as the target acquisition parameters, and the fields collected by the field acquisition operation performed based on the target acquisition parameters are determined as the first associated fields. A preset field list is constructed based on the first associated field.

[0077] The field processing method provided in this embodiment determines, during each round of field collection, adjusted collection parameters for the next round based on the historical collection parameters of the previous round and the percentage of unused fields in the second associated fields collected in the previous round. This allows collection of the first associated fields to be performed based on these adjusted collection parameters. By adjusting the collection parameters in each round, more field usage scenarios can be covered. Furthermore, by presetting termination conditions, field collection operations can be accurately terminated based on these termination conditions, improving the accuracy and efficiency of constructing the preset field list.

[0078] Further, based on any of the above embodiments, wherein adjusting the collection parameters includes adjusting the collection flow and adjusting the collection duration. Step 304 includes:

[0079] The termination condition includes at least one of the following:

[0080] The adjustment of the collected flow rate reaches a preset flow threshold;

[0081] The adjusted collection duration reaches a preset duration threshold.

[0082] In this embodiment, a flow threshold and / or a duration threshold may be preset. The termination condition may be that the field collection operation is terminated when the collected flow reaches the preset flow threshold and / or the collected duration reaches the preset duration threshold.

[0083] Therefore, in response to adjusting the collection flow to reach a preset flow threshold, and / or adjusting the collection duration to reach a preset duration threshold, the fields in the associated fields that match the usage fields in the preset usage field list are deleted.

[0084] For example, the traffic threshold can be 100%, and the duration threshold can be 24 hours. Therefore, if the adjusted collection traffic for this round of field collection is determined to be 100% and the adjusted collection duration reaches 24 hours, the field collection operation can be stopped. A preset field list is constructed based on the unused fields in the currently collected associated fields.

[0085] The field processing method provided in this embodiment pre-sets a traffic threshold and / or duration threshold, thereby determining that the current field collection operation is complete when the collection traffic reaches the preset traffic threshold and / or the collection duration reaches the preset duration threshold. Performing field collection based on these traffic thresholds and / or duration thresholds generates a preset field list that covers all unused fields corresponding to external services, improving the accuracy of the preset field list.

[0086] Figure 4 A flow chart of a field processing method provided in another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 4 As shown, step 303 includes:

[0087] Step 401: Determine a usage field in the second associated field based on the usage field list.

[0088] Step 402: Delete the used field in the second associated field to obtain a second unused field in the second associated field.

[0089] Step 403: Determine the proportion of the second unused field in the second associated field.

[0090] Step 404: Determine the adjusted acquisition parameters based on the historical acquisition parameters and the proportion information.

[0091] In this embodiment, during each round of field collection operation, the adjusted collection parameters for the current round of field collection operation are determined based on the historical collection parameters of the previous round and the ratio of unused fields in the second associated fields collected in the previous round.

[0092] Optionally, for the second associated fields collected in the previous round of field collection, the used fields in the second associated fields can be determined using the used field list. The used fields in the second associated fields collected in the previous round of field collection are deleted to obtain the unused fields in the second associated fields collected in the previous round of field collection. Therefore, the ratio of the unused fields to all the second associated fields can be calculated to determine the proportion of the unused fields in the second associated fields. Adjustment of the collection parameters is determined based on the historical collection parameters and the proportion information.

[0093] The field processing method provided in this embodiment determines the adjusted collection parameters of the current round of field collection operations based on the historical collection parameters of the previous round and the proportion of unused fields in the second associated fields collected in the previous round during each round of field collection operations. Therefore, different adjusted collection operations can be used for field collection during each round of field collection operations, thereby covering more usage scenarios of the fields and improving the accuracy and comprehensiveness of the preset field list.

[0094] Further, based on any of the above embodiments, the adjusting of the collection parameters includes adjusting the collection flow and adjusting the collection duration, and the historical collection parameters include historical collection flow and historical collection duration. Step 404 includes:

[0095] The adjusting of the collected flow includes: a sum of a first product and the historical collected flow, wherein the first product is a product of the historical collected flow and the proportion information;

[0096] The adjusted collection duration includes: the sum of a second product and the historical collection duration, where the second product is the product of the historical collection duration and the proportion information.

[0097] In this embodiment, adjusting the collection parameters includes adjusting the collection flow rate and adjusting the collection duration, and the historical collection parameters include the historical collection flow rate and the historical collection duration. The current adjusted collection flow rate can be determined based on the historical collection flow rate and the percentage of unused fields in the second associated field collected in the previous round, and the current adjusted collection duration can be determined based on the historical collection duration of the previous round and the percentage of unused fields in the second associated field collected in the previous round.

[0098] Optionally, a first product of the historical collected traffic and the proportion information may be calculated, and the sum of the first product and the historical collected traffic may be determined as the adjusted collected traffic.

[0099] The second product of the historical collection duration and the proportion information is calculated, and the sum of the second product and the historical collection duration is determined as the adjusted collection duration.

[0100] By using the above method to determine the adjusted collection parameters, the adjusted collection parameters for each round can be slightly larger than the historical collection parameters for the previous round. Therefore, more field usage scenarios can be covered in each round of collection operations.

[0101] The field processing method provided in this embodiment determines the current adjusted collection traffic through the historical collection traffic of the previous round and the proportion of unused fields in the second associated fields collected in the previous round, and determines the current adjusted collection duration through the historical collection duration of the previous round and the proportion of unused fields in the second associated fields collected in the previous round. Therefore, different adjusted collection operations can be used for field collection in each round of field collection operation, thereby covering more usage scenarios of the field and improving the accuracy and comprehensiveness of the preset field list.

[0102] Figure 5 A flow chart of a field processing method provided in another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 5 As shown, the method further includes:

[0103] Step 501: Perform a first round of field collection operation based on a data request associated with an external service according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection flow and initial collection duration.

[0104] Step 502: Determine all fields used by the external service in the initial field.

[0105] Step 503: Construct the used field list based on all the fields.

[0106] In this embodiment, in order to accurately implement the identification operation of unused fields, a used field list may be constructed first, wherein the used field list may include all fields used by external services.

[0107] Optionally, a field collection operation may be performed on associated fields corresponding to all data requests associated with multiple external services according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection traffic and initial collection duration.

[0108] For example, initial field collection can be performed while the service is running. To minimize the impact of initial field collection on system services, field collection can be performed with smaller initial collection parameters. For example, the initial collection rate can be 1%, and the initial collection duration can be one hour. Therefore, the associated fields of all requests to the external service can be collected at 1% of the collection rate. After one hour of collection, all collected fields are determined as initial fields.

[0109] Furthermore, all fields used by the multiple external services in the initial fields may be determined, and a used field list may be constructed based on all fields used by the multiple external services.

[0110] The field processing method provided in this embodiment pre-collects the initial fields using initial collection parameters. This allows the determination of all fields used by multiple external services based on the initial fields, and the construction of a list of used fields. This allows the subsequent accurate identification of unused fields within the associated fields based on this list of used fields.

[0111] Further, based on any of the above embodiments, the field used by the external service is associated with a preset tag, and the preset tag is set for the field by the external service after using the field for data processing. Step 502 includes:

[0112] The said all fields include: all fields having preset marks in the said initial fields.

[0113] In this embodiment, after using a field for data processing, the external service can set a preset tag for the field. For example, the field's associated count information is updated. Therefore, each field used by the external service is associated with a preset tag.

[0114] Therefore, after the initial fields are collected according to the initial collection parameters, the use fields can be identified based on the preset tags associated with the fields.

[0115] Optionally, all fields with preset marks may be determined in the initial field, and all fields used by multiple external services may be determined based on all fields with preset marks.

[0116] The field processing method provided in this embodiment sets a preset tag for the field when using the field for data processing, so that the use field can be accurately identified based on the preset tag, thereby improving the efficiency and accuracy of constructing the use field list.

[0117] Figure 6 A structural diagram of a field processing device provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the device includes: an acquisition module 61, a determination module 62, a deletion module 63, and a processing module 64, wherein the acquisition module 61 is used to obtain multiple to-be-processed fields corresponding to data requests associated with preset external services. The determination module 62 is used to determine multiple target fields in the to-be-processed fields that match a preset field list, wherein the preset field list includes the first unused field associated with each external service, and the preset field list is obtained by deleting the first associated fields of the external services collected based on target collection parameters based on a preset used field list. The deletion module 63 is used to perform a deletion operation on the target fields to obtain the remaining fields. The processing module 64 is used to perform data processing operations based on the remaining fields.

[0118] Furthermore, based on any of the above embodiments, the apparatus further includes: a collection module configured to, after constructing the used field list, perform at least one round of field collection operations on the data request to obtain first associated fields, wherein the used field list is constructed based on initial fields collected in the first round of field collection operations; and a determination module configured to determine the preset field list based on the first associated fields and the used field list.

[0119] Further, based on any of the above embodiments, the determination module includes: a first determination unit, configured to determine the historical collection parameters corresponding to the previous round of field collection operations. A second determination unit, configured to determine the second associated field collected by the previous round of field collection operations. A calculation unit, configured to determine the adjustment collection parameters based on the historical collection parameters and the second associated fields. A collection unit, configured to perform the next round of field collection operations according to the adjusted collection parameters. A processing unit, configured to determine the adjusted collection parameters as the target collection parameters in response to determining that the adjusted collection parameters meet the preset termination conditions, and to determine the fields collected in the next round of field collection operations performed based on the target collection parameters as the first associated fields. A deletion unit, configured to perform a deletion operation on the fields in the first associated fields that match the used fields in the used field list to obtain the preset field list.

[0120] Further, based on any of the above embodiments, the calculation unit includes: a determination subunit configured to determine a used field in the second associated field based on the used field list; a deletion subunit configured to delete a used field in the second associated field to obtain a second unused field in the second associated field; a calculation subunit configured to determine a proportion of the second unused field in the second associated field; and a processing subunit configured to determine the adjusted collection parameter based on the historical collection parameter and the proportion information.

[0121] Further, based on any of the above embodiments, the adjusting of the collection parameters includes adjusting the collection flow rate and adjusting the collection duration, and the historical collection parameters include the historical collection flow rate and the historical collection duration. The adjusting of the collection flow rate includes: the sum of a first product and the historical collection flow rate, where the first product is the product of the historical collection flow rate and the percentage information. The adjusting of the collection duration includes: the sum of a second product and the historical collection duration, where the second product is the product of the historical collection duration and the percentage information.

[0122] Furthermore, based on any of the above embodiments, wherein adjusting the collection parameters includes adjusting the collection flow rate and adjusting the collection duration, and the termination condition includes at least one of the following: the adjusted collection flow rate reaches a preset flow rate threshold, or the adjusted collection duration reaches a preset duration threshold.

[0123] Furthermore, based on any of the above embodiments, the apparatus further includes: a collection module configured to perform a first round of field collection based on data requests associated with the external service according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection traffic and initial collection duration; a determination module configured to determine all fields used by the external service in the initial fields; and a construction module configured to construct the used field list based on all fields.

[0124] Furthermore, based on any of the above embodiments, the fields used by the external service are associated with a preset tag, wherein the preset tag is set by the external service for the field after the field is used for data processing. The all fields include: all fields in the initial field that have the preset tag.

[0125] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0126] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, including:

[0127] at least one processor; and

[0128] a memory communicatively connected to the at least one processor; wherein,

[0129] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any one of the above embodiments.

[0130] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method described in any one of the above embodiments.

[0131] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0132] Figure 7Schematic diagram of an electronic device provided for an embodiment of the present disclosure. Electronic device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0133] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0134] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0135] The computing unit 701 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the field processing method. For example, in some embodiments, the field processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the field processing method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the field processing method by any other appropriate means (e.g., by means of firmware).

[0136] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0140] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0141] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0142] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0143] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A field processing method, comprising: Get the pending fields corresponding to the data requests associated with multiple preset external services; Determining a plurality of target fields among the to-be-processed fields that match a preset field list, wherein the preset field list includes a first unused field associated with each of the external services, the preset field list being obtained by deleting the first associated fields of the external services collected based on the target collection parameters based on a preset used field list; Perform a deletion operation on the target field to obtain the remaining fields; performing a data processing operation based on the remaining fields; The method further includes, after constructing the used field list, performing at least one round of field collection operations on the data request to obtain a first associated field, wherein the used field list is constructed based on initial fields collected in the first round of field collection operations; Determine the historical collection parameters corresponding to the previous round of field collection operations; Determine the second associated field collected in the previous round of field collection operation; Determining an adjusted acquisition parameter based on the historical acquisition parameter and the second associated field; Perform the next round of field collection operations according to the adjusted collection parameters; In response to determining that the adjusted acquisition parameters meet a preset termination condition, the adjusted acquisition parameters are determined as the target acquisition parameters, and the fields collected in the next round of field acquisition operations performed based on the target acquisition parameters are determined as the first associated fields; the target acquisition parameters include acquisition flow rate and acquisition duration; the termination condition includes at least one of the following: the adjusted acquisition flow rate reaches a preset flow rate threshold; the adjusted acquisition duration reaches a preset duration threshold; A deletion operation is performed on the fields in the first associated fields that match the used fields in the used field list to obtain the preset field list.

2. The method according to claim 1, wherein The determining and adjusting the collection parameter based on the historical collection parameter and the second associated field includes: determining a usage field in the second associated field based on the usage field list; Deleting a used field in the second associated field to obtain a second unused field in the second associated field; Determine the proportion of the second unused field in the second associated field; The adjusted acquisition parameters are determined based on the historical acquisition parameters and the proportion information.

3. The method according to claim 2, wherein: The adjusting of the acquisition parameters includes adjusting the acquisition flow and adjusting the acquisition duration, and the historical acquisition parameters include historical acquisition flow and historical acquisition duration; The adjusting of the collected flow includes: a sum of a first product and the historical collected flow, wherein the first product is a product of the historical collected flow and the proportion information; The adjusted collection duration includes: the sum of a second product and the historical collection duration, where the second product is the product of the historical collection duration and the proportion information.

4. The method according to claim 1, wherein The adjusting of the acquisition parameters includes adjusting the acquisition flow and adjusting the acquisition time; The termination condition includes at least one of the following: The adjustment of the collected flow rate reaches a preset flow threshold; The adjusted collection duration reaches a preset duration threshold.

5. The method according to any one of claims 1 to 4, further comprising: Performing a first round of field collection operations based on data requests associated with external services according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection traffic and initial collection duration; Determine all fields used by the external service in the initial field; The used field list is constructed based on all the fields.

6. The method according to claim 5, wherein: The field used by the external service is associated with a preset tag, where the preset tag is set for the field by the external service after the field is used for data processing; The said all fields include: all fields having preset marks in the said initial fields.

7. A field processing device, comprising: An acquisition module is used to obtain the to-be-processed fields corresponding to the data requests associated with multiple preset external services; a determination module, configured to determine a plurality of target fields among the fields to be processed that match a preset field list, wherein the preset field list includes a first unused field associated with each of the external services, the preset field list being obtained by deleting the first associated fields of the external services collected based on the target collection parameters based on a preset used field list; A deletion module, configured to perform a deletion operation on the target field to obtain the remaining fields; a processing module, configured to perform a data processing operation based on the remaining fields; The device further comprises: a collection module, configured to, after constructing a used field list, perform at least one round of field collection operations on the data request to obtain a first associated field, wherein the used field list is constructed based on initial fields collected in the first round of field collection operations; a determining module, configured to determine the preset field list based on the first associated field and the used field list; The determination module includes: A first determining unit, configured to determine historical collection parameters corresponding to a previous round of field collection operations; a second determining unit configured to determine a second associated field collected in a previous round of field collection operations; and a calculating unit configured to determine an adjusted collection parameter based on the historical collection parameters and the second associated field. A collection unit, configured to perform a next round of field collection operations according to the adjusted collection parameters; a processing unit configured to, in response to determining that the adjusted collection parameter satisfies a preset termination condition, determine the adjusted collection parameter as the target collection parameter, and determine the field collected in the next round of field collection operation performed based on the target collection parameter as the first associated field; the target collection parameter includes a collection flow rate and a collection duration; the termination condition includes at least one of the following: the adjusted collection flow rate reaches a preset flow rate threshold; the adjusted collection duration reaches a preset duration threshold; The deleting unit is configured to perform a deleting operation on the fields in the first associated fields that match the used fields in the used field list to obtain the preset field list.

8. The device according to claim 7, wherein The calculation unit includes: a determining subunit, configured to determine a usage field in the second associated field based on the usage field list; a deleting subunit, configured to delete a used field in the second associated field, and obtain a second unused field in the second associated field; a calculation subunit, configured to determine a proportion of the second unused field in the second associated field; The processing subunit is configured to determine the adjusted acquisition parameter based on the historical acquisition parameter and the proportion information.

9. The device according to claim 8, wherein The adjusting of the acquisition parameters includes adjusting the acquisition flow and adjusting the acquisition duration, and the historical acquisition parameters include historical acquisition flow and historical acquisition duration; The adjusting of the collected flow includes: a sum of a first product and the historical collected flow, wherein the first product is a product of the historical collected flow and the proportion information; The adjusted collection duration includes: the sum of a second product and the historical collection duration, where the second product is the product of the historical collection duration and the proportion information.

10. The device according to claim 7, wherein The adjusting of the acquisition parameters includes adjusting the acquisition flow and adjusting the acquisition time; The termination condition includes at least one of the following: The adjustment of the collected flow rate reaches a preset flow threshold; The adjusted collection duration reaches a preset duration threshold.

11. The device according to any one of claims 7 to 10, further comprising: A collection module, configured to perform a first round of field collection operations based on data requests associated with external services according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection flow and initial collection duration; a determination module, configured to determine all fields used by the external service in the initial field; A construction module is used to construct the used field list based on all the fields.

12. The device according to claim 11, wherein The field used by the external service is associated with a preset tag, where the preset tag is set for the field by the external service after the field is used for data processing; The said all fields include: all fields having preset marks in the said initial fields.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

Images