Field processing method and device, equipment, computer readable storage medium and product

Abstract

The invention provides a field processing method, device and equipment, a computer readable storage medium and a product, and relates to the field of data processing, in particular to the field of big data. According to the specific implementation scheme, the method comprises the steps of obtaining to-be-processed fields corresponding to data requests associated with a plurality of preset external services; a plurality of target fields, matched with a preset field list, in the to-be-processed fields are determined, the preset field list comprises first unused fields associated with the external services, and the preset field list is obtained by deleting first associated fields of the external services collected based on the target collection parameters based on a preset used field list; executing deletion operation on the target field to obtain a residual field; a data processing operation is performed based on the remaining fields. Therefore, the field offline operation can be guided based on the preset field list, and the system performance problem caused by full field collection is avoided.

Description

Technical Field

[0001] The present disclosure relates to big data in data processing, and more particularly to a field processing method, apparatus, device, computer-readable storage medium and product. Background Art

[0002] In modern backend service architectures, systems often need to obtain data from multiple external services or APIs. These data are stored in the form of structure fields to support business logic and application functions.

[0003] With the continuous iteration of product business, the number of fields requested and stored by the system continues to increase. This expansion of fields may lead to a decline in system performance, and too many fields will increase learning costs, reduce iteration efficiency, and even affect system stability. Summary of the invention

[0004] The present disclosure provides a field processing method, apparatus, device, computer-readable storage medium and product for guiding field offline operations to avoid system performance problems caused by full field collection.

[0005] According to a first aspect of the present disclosure, a field processing method is provided, including:

[0006] Get the pending fields corresponding to the data requests associated with multiple preset external services;

[0007] Determine a plurality of target fields in the to-be-processed fields that match a preset field list, wherein the preset field list includes a first unused field associated with each of the external services, and the preset field list is obtained by deleting the first associated field of the external service collected based on the target collection parameter based on a preset used field list;

[0008] Perform a deletion operation on the target field to obtain remaining fields;

[0009] A data processing operation is performed based on the remaining fields.

[0010] According to a second aspect of the present disclosure, a field processing device is provided, including:

[0011] An acquisition module, used to acquire the to-be-processed fields corresponding to the data requests associated with multiple preset external services;

[0012] A determination module, used to determine a plurality of target fields in the to-be-processed fields that match a preset field list, wherein the preset field list includes a first unused field associated with each of the external services, and the preset field list is obtained by deleting the first associated field of the external service collected based on the target collection parameter based on a preset used field list;

[0013] A deletion module, used to perform a deletion operation on the target field to obtain remaining fields;

[0014] A processing module is used to perform data processing operations based on the remaining fields.

[0015] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0021] The technology disclosed in the present invention solves the problem of system performance degradation caused by full data collection for multiple external services.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 A diagram of the system architecture on which the present disclosure is based;

[0025] Figure 2 A flowchart of a field processing method provided in an embodiment of the present disclosure;

[0026] Figure 3 A flowchart of a field processing method provided in yet another embodiment of the present disclosure;

[0027] Figure 4 A flowchart of a field processing method provided in yet another embodiment of the present disclosure;

[0028] Figure 5 A flowchart of a field processing method provided in yet another embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of the structure of a field processing device provided in an embodiment of the present disclosure;

[0030] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] The present disclosure provides a field processing method, device, equipment, computer-readable storage medium and product, which are applied to big data in the field of data processing to identify and guide offline useless fields, realize efficient field management and optimize system performance.

[0033] In modern backend service architectures, systems usually need to communicate with multiple external services to obtain data from them for subsequent business processing. For example, in order to accurately recommend content, recommendation systems need to obtain forward data, reverse data, resource granularity data, historical data, and other data from multiple external services.

[0034] As product services continue to evolve, the number of fields requested and stored by the system continues to increase. This expansion of fields may lead to a decline in system performance, and too many fields will increase learning costs, reduce iteration efficiency, and even affect system stability.

[0035] In related technologies, all data requests associated with external services are generally collected in full. However, with the continuous iteration of product business, the number of fields requested and stored by the system continues to increase. Full data collection often leads to a decline in system performance, and too many fields will increase learning costs, reduce iteration efficiency, and even affect system stability.

[0036] In the process of solving the above technical problems, the inventors found through research that there are a lot of data associated with external services, among which there are often some fields that are not used by external services. Therefore, in order to accurately obtain available data, the fields in the service can be collected in advance, and a preset field list consisting of all unused data can be constructed. Therefore, in the subsequent data acquisition process, the preset field list can be used to guide the offline of useless fields. In this way, only the data that needs to be used can be acquired. It effectively reduces the amount of data that needs to be acquired and stored, and realizes efficient field management and optimization of system performance. It can not only reduce the waste of system resources, but also improve development and maintenance efficiency, ensuring that the system remains stable and efficient in a rapidly changing business environment.

[0037] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0038] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 1-Figure 7 The embodiments of the present disclosure are further refined.

[0039] Figure 1 This is the system architecture diagram on which the present disclosure is based, such as Figure 1 As shown, the system architecture based on the present disclosure includes at least a server 11 and a plurality of external servers 12. The server 11 is respectively connected to each external server for communication.

[0040] Based on the above system architecture, after obtaining the to-be-processed fields corresponding to all data requests associated with the external server 12, the server 11 can offline the unused fields in the to-be-processed fields based on the pre-built preset field list, so as to achieve efficient field management and optimization of system performance.

[0041] Figure 2 A flow chart of a field processing method provided in an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the method includes:

[0042] Step 201: Obtain to-be-processed fields corresponding to data requests associated with multiple preset external services.

[0043] The execution subject of this embodiment is a field processing device. The field processing device can be coupled to a server. The server can be a server associated with a preset system. For example, it can be a server associated with a recommendation system. The server can be connected to multiple external services.

[0044] In this embodiment, the server can obtain the to-be-processed fields corresponding to all data requests associated with multiple preset external services, so that data processing operations can be performed based on the to-be-processed fields later.

[0045] For example, the field processing device can be coupled to a server associated with the recommendation system. In order to implement content recommendation operations, the recommendation system needs to obtain forward data, reverse data, resource granularity data, historical data and other data from multiple external services. In this way, data processing can be performed based on the above data to implement accurate content recommendation operations.

[0046] As the business continues to iterate, the number of fields requested and stored by the recommendation system continues to increase. The expansion of fields may lead to a decline in system performance, and too many fields will increase learning costs, reduce iteration efficiency, and even affect system stability. Therefore, after obtaining the pending fields corresponding to all data requests associated with multiple preset external services, you can also delete the fields that are not used by external services in the pending fields.

[0047] Step 202: determine multiple target fields in the to-be-processed fields that match a preset field list, wherein the preset field list includes the first unused field associated with each of the external services, and the preset field list is obtained by deleting the first associated field of the external service collected based on the target collection parameters based on a preset used field list.

[0048] In this embodiment, in order to implement offline guidance operation for fields not used by external services, a preset field list may be pre-constructed, wherein the preset field list pre-stores the first unused field associated with each external service.

[0049] Optionally, a list of used fields can be pre-built by combining all used fields of multiple external services in the data request process. Therefore, after obtaining the first associated field corresponding to the full request associated with multiple external services, the used fields in the first associated field can be deleted based on the list of used fields. A preset field list is constructed based on the remaining fields after deletion.

[0050] Therefore, after obtaining the pending fields corresponding to all data requests associated with multiple preset external services, multiple target fields in the pending fields that match the preset field list are determined, and the preset field list pre-stores all unused fields associated with each external service.

[0051] Step 203: Perform a deletion operation on the target field to obtain remaining fields.

[0052] Step 204: Perform data processing operations based on the remaining fields.

[0053] In this embodiment, after determining multiple target fields that are not used by external services in the fields to be processed, in order to avoid the problem of performance degradation of the system caused by the storage of too many fields, the multiple target fields in all the fields to be processed can be deleted. Therefore, after the deletion, the remaining fields in the fields to be processed are the fields used by the external services, and therefore, data processing operations can be performed based on the remaining fields in all the fields to be processed.

[0054] The field processing method provided in this embodiment pre-builds a preset field list including all unused fields of external services, so that after obtaining the fields to be processed from multiple external services, the offline operation of the unused fields in the fields to be processed can be guided based on the preset field list. Therefore, only the used fields in the fields to be processed can be subsequently stored and data processed. Therefore, efficient management of the fields to be processed can be achieved, and the efficiency of data processing is improved.

[0055] Further, based on any of the above embodiments, the method further includes:

[0056] After constructing the used field list, at least one round of field collection operation is performed on the data request to obtain a first associated field, wherein the used field list is constructed based on the initial fields collected by the first round of field collection operation.

[0057] The preset field list is determined based on the first associated field and the used field list.

[0058] In this embodiment, in order to implement offline guidance operations for fields not used by external services, a preset field list may be pre-constructed, wherein all unused fields associated with each external service are pre-stored in the preset field list.

[0059] In order not to affect the system service, the preset field list can be constructed during the operation of the system service.

[0060] Optionally, multiple rounds of field collection operations may be performed on all data requests associated with multiple external requests. A preset field list is determined based on the first associated fields collected through the multiple rounds of field collection operations and the used field list.

[0061] Among them, firstly, the first round of field collection operation can be performed according to the initial collection parameters. A used field list is constructed based on the initial fields collected by the first round of field collection operation. Therefore, after the used field list is constructed, at least one round of field collection operation can be performed on the data request to obtain the first associated field. By pre-constructing the used field list, in each subsequent round of field collection operation, the used fields in the collected second associated fields can be deleted based on the used field list to obtain unused fields.

[0062] It should be noted that in order to avoid the impact of the construction operation of the preset field list on the system service, during the initial collection process, the field collection operation can be performed according to a smaller collection flow and collection duration. For example, all fields can be collected according to a collection flow of 1% of the full data request, and the collection duration can be set to one hour. During each round of field collection operation, the collection parameters are continuously adjusted and expanded to cover more field usage scenarios. Until the collection flow reaches 100% and the collection duration reaches a full day, the field collection operation is completed, and a preset field list is built based on the unused fields in the first associated field currently collected.

[0063] The field processing method provided in this embodiment performs multiple rounds of field collection operations on all data requests associated with multiple external requests in advance, thereby determining a preset field list based on the collected first associated fields and the use field list. Therefore, the offline operation of the field can be accurately guided based on the preset field list, thereby improving the accuracy of field management and the efficiency of field-based data processing.

[0064] Figure 3 A flowchart of a field processing method provided by another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 3 As shown, the determining the preset field list based on the first associated field and the used field list includes:

[0065] Step 301: Determine the historical collection parameters corresponding to the previous round of field collection operations.

[0066] Step 302: Determine the second associated field collected in the previous round of field collection operation.

[0067] Step 303: Determine and adjust collection parameters based on the historical collection parameters and the second associated field.

[0068] Step 304: Execute the next round of field collection operation according to the adjusted collection parameters.

[0069] Step 305: In response to determining that the adjusted acquisition parameter meets a preset termination condition, the adjusted acquisition parameter is determined as the target acquisition parameter, and the field acquired in the next round of field acquisition operation performed based on the target acquisition parameter is determined as the first associated field.

[0070] Step 306: Delete the fields in the first associated fields that match the used fields in the used field list to obtain the preset field list.

[0071] In this embodiment, multiple rounds of field collection operations may be performed on all data requests associated with multiple external requests. In each round of field collection operation, collection parameters are continuously adjusted and expanded to cover more field usage scenarios.

[0072] Optionally, in order to cover more field usage scenarios, the adjusted acquisition parameters of this round can be determined based on the historical acquisition parameters of the previous round and the proportion of unused fields in the second associated fields collected in the previous round. Therefore, the historical acquisition parameters corresponding to the previous round of field acquisition operations can be determined, as well as the second associated fields collected in the previous round of field acquisition operations. The adjusted acquisition parameters are determined based on the historical acquisition parameters and the second associated fields. The next round of field acquisition operations are performed on the second associated fields of multiple external services according to the adjusted acquisition parameters.

[0073] It should be noted that before constructing the preset field list, it is necessary to first construct the use field list. The initial field can be collected according to the preset initial parameters, and all the use fields can be identified in the initial field to construct the use field list. Therefore, the collection operation of the initial field is the first round of field collection operation. The multiple rounds of field collection operations when constructing the preset field list in the future all have the previous round of field collection operations.

[0074] Furthermore, in order to accurately construct the preset field list, a termination condition may be preset. After determining the adjustment acquisition parameters of this round and completing the field acquisition operation based on the adjustment acquisition parameters, it may be determined whether the adjustment acquisition parameters meet the termination condition.

[0075] In response to the adjustment acquisition parameter satisfying the preset termination condition, that is, the current field acquisition operation has ended, the current adjustment acquisition parameter can be determined as the target acquisition parameter, and the field acquired by the field acquisition operation performed based on the target acquisition parameter is determined as the first associated field. A preset field list is constructed based on the first associated field. Among them, the fields in the first associated field that match the use fields in the preset use field list can be deleted to obtain the preset field list.

[0076] In response to the adjustment of the acquisition parameters not meeting the preset termination conditions, it indicates that the field acquisition operation has not ended, and the next round of field acquisition operations needs to be further performed. At this time, the current field acquisition operation can be determined as the previous round of field acquisition operations, that is, the current adjustment acquisition parameters are determined as historical acquisition parameters, and the second associated field collected in this round is determined as the second associated field collected in the previous round. Return to the step of determining the historical acquisition parameters corresponding to the previous round of field acquisition operations. After multiple cycles of acquisition, when it is determined that the current adjustment acquisition parameters meet the preset termination conditions, the current adjustment acquisition parameters are determined as the target acquisition parameters, and the fields collected by the field acquisition operation performed based on the target acquisition parameters are determined as the first associated fields. Construct a preset field list based on the first associated field.

[0077] The field processing method provided in this embodiment determines the adjusted acquisition parameters of the next round of field acquisition operation based on the historical acquisition parameters of the previous round and the proportion of unused fields in the second associated fields collected in the previous round during each round of field acquisition operation, so that the acquisition operation of the first associated field can be implemented based on the adjusted acquisition parameters. By adjusting the acquisition parameters in each round, more usage scenarios of the field can be covered. In addition, by pre-setting the termination condition, the termination of the field acquisition operation can be accurately implemented based on the termination condition, thereby improving the accuracy and efficiency of the preset field list construction.

[0078] Further, based on any of the above embodiments, the adjusting of the collection parameters includes adjusting the collection flow and adjusting the collection duration. Step 304 includes:

[0079] The termination condition includes at least one of the following:

[0080] The adjusting the collected flow rate reaches a preset flow rate threshold;

[0081] The adjusted collection duration reaches a preset duration threshold.

[0082] In this embodiment, a flow threshold and / or a duration threshold may be preset. The termination condition may be to terminate the field collection operation when the collected flow reaches a preset flow threshold and / or the collected duration reaches a preset duration threshold.

[0083] Therefore, in response to adjusting the collected traffic to reach a preset traffic threshold, and / or adjusting the collected duration to reach a preset duration threshold, a deletion operation is performed on the fields in the associated fields that match the used fields in the preset used field list.

[0084] Taking actual application as an example, the flow threshold can be 100%, and the duration threshold can be 24 hours. Therefore, when the adjusted collection flow of the current round of field collection operation is determined to be 100% and the adjusted collection duration reaches 24 hours, the field collection operation can be stopped. A preset field list is constructed based on the unused fields in the currently collected associated fields.

[0085] The field processing method provided in this embodiment pre-sets a flow threshold and / or a duration threshold, so that when the collection flow is adjusted to reach the preset flow threshold and / or the collection duration is adjusted to reach the preset duration threshold, it is determined that the current field collection operation is completed. The field collection operation is performed based on the flow threshold and / or duration threshold, and the preset field list that can be generated can cover all unused fields corresponding to the external service, thereby improving the accuracy of the preset field list.

[0086] Figure 4 A flowchart of a field processing method provided by another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 4 As shown, step 303 includes:

[0087] Step 401: Determine a usage field in the second associated field based on the usage field list.

[0088] Step 402: Delete the used field in the second associated field to obtain a second unused field in the second associated field.

[0089] Step 403: Determine the proportion of the second unused field in the second associated field.

[0090] Step 404: Determine the adjusted acquisition parameters based on the historical acquisition parameters and the proportion information.

[0091] In this embodiment, during each round of field collection operation, the adjusted collection parameters of the current round of field collection operation are determined based on the historical collection parameters of the previous round and the proportion information of unused fields in the second associated fields collected in the previous round.

[0092] Optionally, for the second associated fields collected in the previous round of field collection operations, the used fields in the second associated fields can be determined by using a field list. The used fields in the second associated fields collected in the previous round of field collection operations are deleted to obtain unused fields in the second associated fields collected in the previous round of field collection operations. Therefore, the ratio between the unused fields and all the second associated fields can be calculated to determine the proportion of the unused fields in the second associated fields. The collection parameters are adjusted based on the historical collection parameters and the proportion information.

[0093] The field processing method provided in this embodiment determines the adjustment collection parameters of the current round of field collection operations based on the historical collection parameters of the previous round and the proportion of unused fields in the second associated fields collected in the previous round during each round of field collection operations. Therefore, different adjustment collection operations can be used to perform field collection during each round of field collection operations, thereby covering more usage scenarios of the fields and improving the accuracy and comprehensiveness of the preset field list.

[0094] Further, based on any of the above embodiments, the adjusting of the collection parameters includes adjusting the collection flow and adjusting the collection duration, and the historical collection parameters include the historical collection flow and the historical collection duration. Step 404 includes:

[0095] The adjusting of the collected flow includes: a sum of a first product and the historical collected flow, wherein the first product is a product of the historical collected flow and the proportion information;

[0096] The adjusted collection duration includes: the sum of the second product and the historical collection duration, where the second product is the product of the historical collection duration and the proportion information.

[0097] In this embodiment, adjusting the collection parameters includes adjusting the collection flow and adjusting the collection duration, and the historical collection parameters include the historical collection flow and the historical collection duration. The current adjusted collection flow can be determined based on the historical collection flow and the proportion of unused fields in the second associated field collected in the previous round, and the current adjusted collection duration can be determined by the historical collection duration of the previous round and the proportion of unused fields in the second associated field collected in the previous round.

[0098] Optionally, a first product of the historical collected flow and the proportion information may be calculated, and the sum of the first product and the historical collected flow is determined as the adjusted collected flow.

[0099] The second product of the historical collection duration and the proportion information is calculated, and the sum of the second product and the historical collection duration is determined as the adjusted collection duration.

[0100] By adopting the above method to determine the adjusted acquisition parameters, the adjusted acquisition parameters of each round can be slightly larger than the historical acquisition parameters of the previous round, so that more field usage scenarios can be covered in each round of acquisition operations.

[0101] The field processing method provided in this embodiment determines the current adjusted collection traffic through the historical collection traffic of the previous round and the proportion of unused fields in the second associated fields collected in the previous round, and determines the current adjusted collection duration through the historical collection duration of the previous round and the proportion of unused fields in the second associated fields collected in the previous round, so that different adjustment collection operations can be used to perform field collection in each round of field collection operation, thereby covering more usage scenarios of the field and improving the accuracy and comprehensiveness of the preset field list.

[0102] Figure 5 A flowchart of a field processing method provided by another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 5 As shown, the method also includes:

[0103] Step 501: Perform a first round of field collection operation based on a data request associated with an external service according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection flow and initial collection duration.

[0104] Step 502: Determine all fields in the initial field that are used by the external service.

[0105] Step 503: construct the used field list based on all the fields.

[0106] In this embodiment, in order to accurately implement the identification operation of unused fields, a used field list may be constructed first, wherein the used field list may include all fields used by external services.

[0107] Optionally, a field collection operation may be performed on associated fields corresponding to all data requests associated with multiple external services according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection traffic and initial collection duration.

[0108] For example, the initial field collection operation can be performed during the service operation. In order to avoid the impact of the initial field collection operation on the system service, the field collection operation can be performed according to smaller initial collection parameters. For example, the initial collection flow can be 1%, and the initial collection duration can be one hour. Therefore, the collection operation can be performed on the associated fields of the full amount of requests for external services according to the 1% collection flow, and after the collection operation lasts for one hour, all the collected fields are determined as the initial fields.

[0109] Furthermore, all fields used by the multiple external services in the initial fields may be determined, and a used field list is constructed based on all fields used by the multiple external services.

[0110] The field processing method provided in this embodiment uses the initial collection parameters to perform the initial field collection operation in advance, so that all fields used by multiple external services can be determined based on the initial fields, and then a list of used fields can be constructed. Therefore, the identification operation of unused fields in the associated fields can be accurately implemented based on the list of used fields.

[0111] Further, based on any of the above embodiments, the field used by the external service is associated with a preset tag, and the preset tag is set for the field by the external service after using the field for data processing. Step 502 includes:

[0112] The said all fields include: all fields having preset marks in the said initial fields.

[0113] In this embodiment, after using the field for data processing, the external service may set a preset tag for the field, for example, updating the count information associated with the field, etc. Therefore, each field used by the external service is associated with a preset tag.

[0114] Therefore, after the initial fields are collected according to the initial collection parameters, the identification operation of the used fields can be implemented based on the preset tags associated with the fields.

[0115] Optionally, all fields with preset marks can be determined in the initial field. All fields used by multiple external services are determined based on all fields with preset marks.

[0116] The field processing method provided in this embodiment sets a preset tag for the field when using the field for data processing, so that the identification operation of the used field can be accurately implemented based on the preset tag, thereby improving the efficiency and accuracy of constructing the used field list.

[0117] Figure 6 A schematic diagram of the structure of a field processing device provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the device includes: an acquisition module 61, a determination module 62, a deletion module 63 and a processing module 64, wherein the acquisition module 61 is used to obtain the to-be-processed fields corresponding to the data requests associated with multiple preset external services. The determination module 62 is used to determine multiple target fields in the to-be-processed fields that match the preset field list, wherein the preset field list includes the first unused field associated with each of the external services, and the preset field list is obtained by deleting the first associated field of the external service collected based on the target collection parameter based on the preset used field list. The deletion module 63 is used to perform a deletion operation on the target field to obtain the remaining field. The processing module 64 is used to perform a data processing operation based on the remaining field.

[0118] Further, based on any of the above embodiments, the device further includes: a collection module, configured to perform at least one round of field collection operation on the data request after constructing the use field list to obtain a first associated field, wherein the use field list is constructed based on the initial fields collected by the first round of field collection operation. A determination module, configured to determine the preset field list based on the first associated field and the use field list.

[0119] Further, on the basis of any of the above embodiments, the determination module includes: a first determination unit, used to determine the historical collection parameters corresponding to the previous round of field collection operations. A second determination unit, used to determine the second associated field collected by the previous round of field collection operations. A calculation unit, used to determine the adjustment collection parameters based on the historical collection parameters and the second associated fields. A collection unit, used to perform the next round of field collection operations according to the adjusted collection parameters. A processing unit, used to determine the adjusted collection parameters as the target collection parameters in response to determining that the adjusted collection parameters meet the preset termination conditions, and determine the fields collected by the next round of field collection operations performed based on the target collection parameters as the first associated fields. A deletion unit, used to perform a deletion operation on the fields in the first associated fields that match the used fields in the used field list to obtain the preset field list.

[0120] Further, based on any of the above embodiments, the calculation unit includes: a determination subunit, configured to determine a used field in the second associated field based on the used field list; a deletion subunit, configured to delete a used field in the second associated field to obtain a second unused field in the second associated field; a calculation subunit, configured to determine a proportion of the second unused field in the second associated field; and a processing subunit, configured to determine the adjusted acquisition parameter based on the historical acquisition parameter and the proportion information.

[0121] Further, based on any of the above embodiments, the adjusting of the collection parameters includes adjusting the collection flow and adjusting the collection duration, and the historical collection parameters include the historical collection flow and the historical collection duration. The adjusting of the collection flow includes: the sum of the first product and the historical collection flow, the first product being the product of the historical collection flow and the proportion information. The adjusting of the collection duration includes: the sum of the second product and the historical collection duration, the second product being the product of the historical collection duration and the proportion information.

[0122] Further, based on any of the above embodiments, wherein the adjusting of the collection parameters includes adjusting the collection flow and adjusting the collection duration. The termination condition includes at least one of the following: the adjusting of the collection flow reaches a preset flow threshold. The adjusting of the collection duration reaches a preset duration threshold.

[0123] Further, based on any of the above embodiments, the device further includes: a collection module, used to perform a first round of field collection operations based on data requests associated with external services according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection traffic and initial collection duration. A determination module, used to determine all fields used by the external service in the initial field. A construction module, used to construct the used field list based on all fields.

[0124] Further, based on any of the above embodiments, the field used by the external service is associated with a preset tag, and the preset tag is set for the field by the external service after using the field for data processing. The all fields include: all fields with preset tags in the initial field.

[0125] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0126] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, including:

[0127] at least one processor; and

[0128] a memory communicatively connected to the at least one processor; wherein,

[0129] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of the above embodiments.

[0130] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method described in any one of the above embodiments.

[0131] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0132] Figure 7A schematic diagram of the structure of an electronic device provided for an embodiment of the present disclosure. The electronic device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0133] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0134] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0135] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as field processing methods. For example, in some embodiments, the field processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the field processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the field processing method in any other appropriate manner (e.g., by means of firmware).

[0136] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0138] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0140] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0141] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0143] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A field processing method, comprising: Get the pending fields corresponding to the data requests associated with multiple preset external services; Determine a plurality of target fields in the to-be-processed fields that match a preset field list, wherein the preset field list includes a first unused field associated with each of the external services, and the preset field list is obtained by deleting the first associated field of the external service collected based on the target collection parameter based on a preset used field list; Perform a deletion operation on the target field to obtain remaining fields; A data processing operation is performed based on the remaining fields.

2. The method according to claim 1, further comprising: After constructing the use field list, performing at least one round of field collection operation on the data request to obtain a first associated field, wherein the use field list is constructed based on the initial fields collected by the first round of field collection operation; The preset field list is determined based on the first associated field and the used field list.

3. The method according to claim 2, wherein: The determining the preset field list based on the first associated field and the used field list includes: Determine the historical collection parameters corresponding to the previous round of field collection operations; Determine the second associated field collected by the previous round of field collection operation; Determine and adjust acquisition parameters based on the historical acquisition parameters and the second associated field; Perform the next round of field collection operation according to the adjusted collection parameters; In response to determining that the adjusted acquisition parameter satisfies a preset termination condition, determining the adjusted acquisition parameter as the target acquisition parameter, and determining the field acquired in the next round of field acquisition operation performed based on the target acquisition parameter as the first associated field; A deletion operation is performed on the fields in the first associated fields that match the used fields in the used field list to obtain the preset field list.

4. The method according to claim 3, wherein: The determining and adjusting the collection parameter based on the historical collection parameter and the second associated field includes: Determine a usage field in the second associated field based on the usage field list; Deleting a used field in the second associated field to obtain a second unused field in the second associated field; Determine the proportion of the second unused field in the second associated field; The adjusted acquisition parameter is determined based on the historical acquisition parameter and the proportion information.

5. The method according to claim 4, wherein: The adjusting of the collection parameters includes adjusting the collection flow and adjusting the collection duration, and the historical collection parameters include the historical collection flow and the historical collection duration; The adjusting of the collected flow includes: a sum of a first product and the historical collected flow, wherein the first product is a product of the historical collected flow and the proportion information; The adjusted collection duration includes: the sum of the second product and the historical collection duration, where the second product is the product of the historical collection duration and the proportion information.

6. The method according to claim 3, wherein: The adjusting of the acquisition parameters includes adjusting the acquisition flow rate and adjusting the acquisition duration; The termination condition includes at least one of the following: The adjusting the collected flow rate reaches a preset flow rate threshold; The adjusted collection duration reaches a preset duration threshold.

7. The method according to any one of claims 1 to 6, further comprising: Performing a first round of field collection operations based on data requests associated with external services according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection flow and initial collection duration; Determine all fields used by the external service in the initial field; The used field list is constructed based on all the fields.

8. The method according to claim 7, wherein: The field used by the external service is associated with a preset tag, and the preset tag is set for the field by the external service after using the field for data processing; The said all fields include: all fields having preset marks in the said initial fields.

9. A field processing device, comprising: An acquisition module, used to acquire the to-be-processed fields corresponding to the data requests associated with multiple preset external services; A determination module, used to determine a plurality of target fields in the to-be-processed fields that match a preset field list, wherein the preset field list includes a first unused field associated with each of the external services, and the preset field list is obtained by deleting the first associated field of the external service collected based on the target collection parameter based on a preset used field list; A deletion module, used to perform a deletion operation on the target field to obtain remaining fields; A processing module is used to perform data processing operations based on the remaining fields.

10. The device according to claim 9, further comprising: A collection module, configured to perform at least one round of field collection operation on the data request after constructing the use field list to obtain a first associated field, wherein the use field list is constructed based on the initial fields collected by the first round of field collection operation; A determination module is used to determine the preset field list based on the first associated field and the used field list.

11. The device according to claim 10, wherein: The determination module comprises: A first determining unit, used to determine the historical collection parameters corresponding to the previous round of field collection operation; A second determining unit, configured to determine a second associated field collected in a previous round of field collection operation; a calculating unit, configured to determine an adjustment collection parameter based on the historical collection parameter and the second associated field; A collection unit, configured to perform a next round of field collection operations according to the adjusted collection parameters; a processing unit, configured to, in response to determining that the adjusted acquisition parameter satisfies a preset termination condition, determine the adjusted acquisition parameter as the target acquisition parameter, and determine the field acquired in the next round of field acquisition operation performed based on the target acquisition parameter as the first associated field; A deleting unit is used to perform a deleting operation on the fields in the first associated fields that match the used fields in the used field list to obtain the preset field list.

12. The device according to claim 11, wherein The computing unit comprises: a determining subunit, configured to determine a usage field in the second associated field based on the usage field list; a deleting subunit, configured to delete a used field in the second associated field, and obtain a second unused field in the second associated field; A calculation subunit, used to determine the proportion of the second unused field in the second associated field; The processing subunit is used to determine the adjusted acquisition parameter based on the historical acquisition parameter and the proportion information.

13. The device according to claim 12, wherein: The adjusting of the collection parameters includes adjusting the collection flow and adjusting the collection duration, and the historical collection parameters include the historical collection flow and the historical collection duration; The adjusting of the collected flow includes: a sum of a first product and the historical collected flow, wherein the first product is a product of the historical collected flow and the proportion information; The adjusted collection duration includes: the sum of the second product and the historical collection duration, where the second product is the product of the historical collection duration and the proportion information.

14. The device according to claim 11, wherein: The adjusting of the acquisition parameters includes adjusting the acquisition flow rate and adjusting the acquisition duration; The termination condition includes at least one of the following: The adjusting the collected flow rate reaches a preset flow rate threshold; The adjusted collection duration reaches a preset duration threshold.

15. The device according to any one of claims 9 to 14, further comprising: A collection module, used to perform a first round of field collection operation based on the data request associated with the external service according to preset initial collection parameters to obtain initial fields, wherein the initial collection parameters include initial collection flow and initial collection duration; A determination module, used for determining all fields used by the external service in the initial field; A construction module is used to construct the used field list based on all the fields.

16. The device according to claim 15, wherein: The field used by the external service is associated with a preset tag, and the preset tag is set for the field by the external service after using the field for data processing; The said all fields include: all fields having preset marks in the said initial fields.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

19. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 8 when executed by a processor.

Images