A data processing method, device and electronic equipment

Abstract

The application discloses a data processing method and device and electronic equipment, and relates to the technical field of communication, to solve the problems of low timeliness and high possibility of error in the existing manual reporting mode of capability data. The method comprises the following steps: acquiring first capability data from a data provider; performing format conversion on the first capability data according to a preset data format to obtain second capability data after conversion; checking the second capability data; and displaying the second capability data that passes the check. In this way, the timeliness of data collection can be improved by automatically collecting capability data from the data provider, and the capability data displayed can be prevented from having large errors by checking the collected capability data, thereby meeting the aggregation demand of existing artificial intelligence (AI) field capability data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and in particular to a data processing method and device and electronic equipment. BACKGROUND

[0002] In the context of the big data era, capability data includes data of various different sources and different structures. At present, the capability data aggregation method usually adopts manual reporting, that is, each capability user or platform provider queries relevant information in a corresponding data monitoring system, fills in a form of a unified template, and then sends the form to an interface person through email. This method uses a unified template and the same format to solve the problem of multi-source heterogeneity of capability data.

[0003] However, the manual reporting method has low timeliness and a high possibility of errors, which is not conducive to real-time monitoring and fine operation of capabilities. SUMMARY

[0004] Embodiments of the present application provide a data processing method, device and electronic equipment to solve the problem of low timeliness and high possibility of errors in the existing manual reporting method of capability data, which is not conducive to real-time monitoring and fine operation of capabilities.

[0005] In a first aspect, the embodiments of the present application provide a data processing method, comprising:

[0006] obtaining first capability data from a data provider;

[0007] performing format conversion on the first capability data according to a preset data format to obtain second capability data after conversion;

[0008] verifying the second capability data;

[0009] displaying the second capability data that passes the verification.

[0010] Optionally, the obtaining of the first capability data from the data provider comprises:

[0011] in the case that the data provider provides a query interface, constructing a capability query request according to a preset query time and frequency, and pre-stored query interface and request method information of the data provider, and sending the capability query request to the query interface of the data provider;

[0012] receiving the first capability data returned by the data provider in response to the query request.

[0013] Optionally, the obtaining of the first capability data from the data provider comprises:

[0014] In a case where the data provider does not provide a query interface, the first capability data pushed by the data provider at an agreed push time and frequency is received.

[0015] Optionally, the method further comprises:

[0016] In a case where it is detected that the data provider does not push capability data at the agreed push time, reminder information is sent to the data provider according to pre-stored contact information of the data provider.

[0017] Optionally, the checking of the second capability data comprises:

[0018] A prediction value of the data provider's current capability data is obtained by using a pre-trained prediction filling model.

[0019] It is checked whether a difference between the second capability data and the prediction value exceeds a predetermined threshold.

[0020] In a case where the difference does not exceed the predetermined threshold, it is determined that the second capability data passes the check.

[0021] Optionally, after the checking of whether the difference between the second capability data and the prediction value exceeds the predetermined threshold, the method comprises:

[0022] In a case where the difference exceeds the predetermined threshold, capability data is re-acquired from the data provider.

[0023] If the capability data is not re-acquired within a preset time or the re-acquired capability data is empty, the prediction value is filled as the capability data of the data provider.

[0024] Alternatively, if the re-acquired third capability data is the same as the second capability data, the third capability data is recorded, and alarm information is sent to the data provider.

[0025] Optionally, the prediction filling model is a weighted combination model of a long short-term memory (LSTM) model and a capability influence factor model, the LSTM model is trained by using historical capability data of the data provider, and the capability influence factor model is trained by using influence factors of the capability data of the data provider.

[0026] Optionally, the obtaining of the prediction value of the data provider's current capability data by using the pre-trained prediction filling model comprises:

[0027] Capability influence factors fed back by the data provider are obtained.

[0028] obtaining a predicted value of current capability data of the data provider by the prediction filling model according to historical capability data and the capability influencing factor.

[0029] In a second aspect, the embodiments of the present application further provide a data processing apparatus, comprising:

[0030] a first obtaining module, configured to obtain first capability data from a data provider;

[0031] a conversion module, configured to perform format conversion on the first capability data according to a preset data format, to obtain second capability data after conversion;

[0032] a verification module, configured to verify the second capability data;

[0033] a display module, configured to display the second capability data that passes the verification.

[0034] Optionally, the first obtaining module comprises:

[0035] a query unit, configured to, in a case where the data provider provides a query interface, construct a capability query request according to a preset query time and frequency, and query interface and request mode information of the data provider stored in advance, and send the capability query request to the query interface of the data provider;

[0036] a first receiving unit, configured to receive first capability data returned by the data provider in response to the query request.

[0037] Optionally, the first obtaining module comprises:

[0038] a second receiving unit, configured to, in a case where the data provider does not provide a query interface, receive the first capability data pushed by the data provider according to an agreed push time and frequency.

[0039] Optionally, the first obtaining module further comprises:

[0040] a sending unit, configured to, in a case where it is detected that the data provider does not push capability data at the agreed push time, send reminder information to the data provider according to contact information of the data provider stored in advance.

[0041] Optionally, the verification module comprises:

[0042] an obtaining unit, configured to obtain a predicted value of current capability data of the data provider by a pre-trained prediction filling model;

[0043] a verification unit, configured to verify whether a difference between the second capability data and the predicted value exceeds a predetermined threshold;

[0044] wherein, in a case where the difference does not exceed the predetermined threshold, it is determined that the second capability data is verified.

[0045] Optionally, the data processing apparatus further comprises:

[0046] a second obtaining module, configured to re-obtain capability data from the data provider in a case where the difference exceeds the predetermined threshold;

[0047] a filling module, configured to fill the predicted value as the capability data of the data provider if the capability data is not re-obtained within a preset time or the re-obtained capability data is empty;

[0048] a recording and sending module, configured to record the third capability data and send an alarm information to the data provider if the re-obtained third capability data is the same as the second capability data.

[0049] Optionally, the prediction filling model is a weighted combination model of a long short-term memory (LSTM) model and a capability influence factor model, the LSTM model is obtained by training historical capability data of the data provider, and the capability influence factor model is obtained by training influence factors of the capability data of the data provider.

[0050] Optionally, the obtaining unit is configured to obtain capability influence factors fed back by the data provider, and obtain a predicted value of current capability data of the data provider according to historical capability data and the capability influence factors by using the prediction filling model.

[0051] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the data processing method when executing the computer program.

[0052] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the data processing method.

[0053] In the embodiment of the present application, the first capability data is acquired from the data provider, the first capability data is converted into a preset data format to obtain second capability data after conversion, the second capability data is verified, and the second capability data that passes the verification is displayed. In this way, the timeliness of data collection can be improved by automatically collecting capability data from the data provider, and the capability data displayed can be avoided from having large errors by verifying the collected capability data, thereby meeting the aggregation requirements of current artificial intelligence (AI) field capability data. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 is a flowchart of the data processing method provided by the embodiments of the present application;

[0056] Figure 2 is a structural diagram of the data processing device provided by the embodiments of the present application;

[0057] Figure 3 is a structural diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0058] The technical solutions of the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0059] In order to make the embodiments of the present application more clear, the related technical knowledge involved in the embodiments of the present application will be introduced as follows:

[0060] Currently, the deployment of artificial intelligence (AI) field capabilities presents a cloud edge combination form. The cloud end capability is deployed on the platform, which can facilitate one-key access and use of various capability users. The advantage of deploying capabilities on the cloud end is that the cloud end has rich computing resources, the capability model training and inference speed is fast, the capability can be uniformly managed, and the repeated development is avoided, thereby avoiding resource waste. The capability model can also be deployed on any computing device on the edge side, which is closer to the user, and can effectively avoid the delay and data leakage caused by data IO transmission and other related problems.

[0061] In the context of the big data era, capability operation data has undergone tremendous changes in terms of total volume, data format, data service methods, and data relationships. Because business data from relevant departments is relatively dispersed, the data volume varies across business systems, and the data security levels differ between the Business, Operation and Management support (BOM) domains, data types from different sources are inconsistent, and even different types of data may exist from the same source, it has become crucial to aggregate this data from different sources and with different structures. This is beneficial for capability optimization, demonstration, and promotion.

[0062] The current method for aggregating capability operation data primarily involves manual reporting. Each capability user or platform provider queries relevant information in their respective data monitoring system, fills it into a standardized Excel spreadsheet, and then sends it via email to the designated contact person. This method uses a unified template and format to address the issue of multi-source and heterogeneous capability operation data. Existing data aggregation methods in other fields mostly target data with the same structure or origin, utilizing unified data acquisition and communication modules to transmit and aggregate the required data.

[0063] Currently, the methods for aggregating operational data in the AI ​​field still have the following problems:

[0064] Firstly, manual reporting is less timely, more prone to errors, and cannot guarantee uploading at specific times. Therefore, this method of collecting capacity operation data cannot achieve real-time monitoring and refined operation of capabilities.

[0065] Secondly, existing data aggregation methods are mostly for data with the same structure, using a uniform data collection method. However, AI capabilities and deployment environments are diverse, and a uniform collection method cannot meet the data aggregation needs of operational data for all AI capabilities.

[0066] Third, existing data aggregation methods rarely consider the accuracy of data transmission, lack effective models to verify the data, and cannot guarantee the timeliness of data upload when problems occur in the data acquisition or transmission modules, which may result in significant errors when displaying the data.

[0067] To address the shortcomings and problems of the existing technical solutions, this application provides a method for aggregating multi-source heterogeneous AI capability operation data, tailored to the actual needs of AI field operational data aggregation. The overall technical solution consists of three parts: data acquisition, data processing, and data display. The data acquisition part primarily provides two data acquisition methods—query and push—for different platforms and devices. The data processing part mainly verifies the acquired data and then processes the verified data into the format required by the data display part. When the data display module initiates a real-time query request, if the data is unavailable, predicted values ​​are used to fill the gap. The data display part is mainly responsible for visualizing the acquired data and supports user interaction, allowing users to query and display real-time capability data.

[0068] The data processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0069] See Figure 1 , Figure 1 This is a flowchart of the data processing method provided in the embodiments of this application, such as... Figure 1 As shown, it includes the following steps:

[0070] Step 101: Obtain the first capability data from the data provider.

[0071] The aforementioned data provider can refer to the party providing capability data, such as a cloud server with unified deployment of capabilities, or computer equipment with distributed capabilities. Capabilities can refer to things like recognition, recommendation, and text classification. The capability data can refer to the number of capability calls and the number of successful calls, specifically such as average execution time and the maximum concurrency per second that the interface can handle.

[0072] Obtaining the first capability data from the data provider can be achieved by querying the data provider or receiving capability data pushed by the data provider. It should be noted that there can be multiple data providers, meaning the capability data can originate from various sources, and corresponding capability data can be obtained from different data providers. Furthermore, different data providers may use different methods to collect capability data; therefore, the appropriate collection method can be used to obtain the corresponding capability data based on the data collection methods of each data provider.

[0073] Specifically, you can first confirm with the data provider whether they can provide a data query interface.

[0074] Optionally, if the data provider provides a query interface, step 101 includes:

[0075] Based on the preset query time and frequency, as well as the pre-stored query interface and request method information of the data provider, a capability query request is constructed and sent to the query interface of the data provider.

[0076] Receive the first capability data returned by the data provider in response to the query request.

[0077] In one implementation, a query method can be used to collect capability data from a data providing platform or device that provides a query interface.

[0078] Specifically, information such as the capability information of the data provider, the capability user information, the capability provider information, interface fields, request methods, and request headers can be pre-stored in a database table. Furthermore, before querying capability data, the query frequency and query time for that capability can be preset. For example, it can be preset to send a query request at 8:00 AM every day, and to query capability information such as the number of calls and the number of successful calls every 24 hours.

[0079] After obtaining the relevant information from the data provider, the corresponding data of its capabilities can be collected and parsed. First, based on the query interface (i.e., interface field), request method, and request header information recorded in the table, a query request can be constructed and sent to the corresponding query interface at the set query time.

[0080] Optionally, the code that sends the query request may be implemented using the computer programming language Python or other programming languages.

[0081] For example, the URL for querying the call volume of capability B on platform A is 'http: / / IP:port / search', the request method is GET, and the request must include the ID of capability B. This ID is provided by the data provider and stored in a table in the database. The query request constructed using Python would be as follows:

[0082] headers={'content-type':"application / json"}

[0083] req=equests.get('http: / / IP:port / search',params={'ID':B},headers=headers)

[0084] After sending a query request, you can read the request data returned by the query interface of the data provider according to the rules provided by the data provider.

[0085] For example, if the return value of capability B above is {"code":200,"message":"ok","data":[{300}], then the status of the request can be determined based on the code value. For example, code:200 indicates that the request was successful, and code:500 indicates that the server encountered a failure. At the same time, the cumulative number of calls to this capability at this time can be obtained by parsing req.text, which is [{300} pointed to by the data value.

[0086] In this way, through this implementation method, data providers that can provide query interfaces can quickly obtain their capability data by using a self-service query method.

[0087] Optionally, if the data provider does not provide a query interface, step 101 includes:

[0088] The first capability data is received from the data provider according to the agreed push time and frequency.

[0089] In one implementation, for data providing platforms or devices that cannot provide query interfaces, the data provider can use a self-pushing method to receive its capability data.

[0090] Specifically, the capability information, capability user information, capability provider information, unique identifier, contact person, and contact information of the data provider can be pre-stored in a database table. The unique identifier can be a unique ID assigned by the system based on the entry time and order. This unique identifier can be synchronized to the data provider, and the push time, frequency, and push format can be agreed upon with them. For example, data such as the previous day's call volume can be pushed every morning at 9:00 AM, and pushed once every 24 hours, with push formats such as txt and jpg.

[0091] After obtaining the information from the aforementioned data provider, the corresponding data of its capabilities can be collected and processed accordingly. Specifically, the capability data pushed by the data provider according to the agreed push time and frequency can be received, and the corresponding program can be called to parse the capability data sent by the other party according to the agreed data push format.

[0092] For example, if the agreed-upon push format is TXT, a TXT parsing program can be called. The TXT content can be pre-defined as follows: {Unique Identifier: 0917-197, Time: 2022-X-XX, Call Count: 10000}. If the agreed-upon format is JPG, it can be parsed using AI capabilities such as Optical Character Recognition (OCR). No restrictions are placed on the capabilities used here.

[0093] In this way, through this implementation method, for data providers that cannot provide query interfaces, their capability data can be quickly obtained by adopting a data provider self-push method.

[0094] Optionally, the method further includes:

[0095] If it is detected that the data provider has failed to push capability data at the agreed push time, a reminder message is sent to the data provider according to the pre-stored contact information of the data provider.

[0096] In one implementation, when dealing with data providers who cannot provide a query interface, the system can also detect whether the other party has sent capability data at the agreed push time. If the other party does not send capability data at the agreed time, that is, does not receive capability data pushed by the data provider at the agreed time, the system can remind the other party to send data by sending an SMS or email based on the contact information of the data provider stored in the table.

[0097] Assuming the contact person for capability C is X, and the contact information is a mobile phone number, then if data from capability C is not received within the agreed time, a reminder message can be sent to the contact person for capability C:

[0098] If request.POST is not True:

[0099] Send_text(X)

[0100] In this way, this implementation method can remind the data provider to push data even if the data provider does not push the capability data or does not receive the capability data pushed by the data provider, so as to ensure the reliability of capability data collection.

[0101] Step 102: Convert the first capability data according to the preset data format to obtain the converted second capability data.

[0102] Since the capability data formats from different sources, i.e. different data providers, may differ, and the formats of different types of capability data may also differ, in order to ensure the aggregation of capability data from different sources and with different structures, it is necessary to parse and adapt the acquired capability data, that is, to perform a unified format conversion on the acquired capability data according to the preset data format.

[0103] For example, for capability B obtained through querying, assuming the capability provider is Company Q, the capability user is Company W, and the number of capability calls is 10,000, the data can be constructed as {Q: {B: {W: 10,000}}}. For capability C obtained through push, assuming the capability provider is Company R, the capability user is Company T, and the number of capability calls is 8,000, the data can be constructed as {R: {C: {T: 8,000}}}.

[0104] Step 103: Verify the second capability data.

[0105] After converting the acquired first capability data into second capability data, the second capability data can be validated. Specifically, the data processing time and frequency can be determined first based on the previously agreed data query or last push time. For example, if all capability pushes and queries are completed by 10:00 AM, validation of all capability data can begin at 10:01 AM. Then, the accuracy of the currently acquired capability data is verified. Specifically, this can involve verifying whether the second capability data is similar to capability data from the same historical period, or whether it is not significantly different from the predicted capability data, to determine if there are any obvious anomalies in the second capability data. If the second capability data is similar to capability data from the same historical period, or not significantly different from the predicted capability data, the validation can be considered successful.

[0106] Optionally, step 103 includes:

[0107] Obtain the predicted values ​​of the pre-trained prediction filling model for the current capability data of the data provider;

[0108] Verify whether the difference between the second capability data and the predicted value exceeds a predetermined threshold;

[0109] If the difference does not exceed the predetermined threshold, the second capability data verification is deemed successful.

[0110] In one implementation, a capability prediction model can be used to verify the accuracy of the second capability data. Specifically, a prediction-filling model can be pre-trained to predict the current capability data of the data provider. This model predicts the current capability data based on the data provider's historical capability data, such as capability data from the past day. After obtaining the second capability data, the predicted current capability data of the data provider can be obtained from the prediction-filling model. The second capability data is compared with the predicted value, and it is determined whether the difference exceeds a predetermined threshold. If it does, the second capability data is deemed inaccurate and fails verification; if it does not exceed the threshold, the second capability data is deemed accurate and reliable and passes verification. The capability prediction model can be a time-series regression model, and the predetermined threshold can be a pre-set verification threshold, which can be determined based on the characteristics of the corresponding capability, the daily usage volume, etc.

[0111] In this way, by using a predictive filling model to verify the accuracy of the acquired capability data, the verification results can be guaranteed to be relatively reliable, thereby ensuring the accuracy of the final displayed capability data.

[0112] Optionally, the prediction filling model is a weighted composite model of a Long Short-Term Memory (LSTM) network model and a capability influencing factor model. The LSTM model is trained using the historical capability data of the data provider, and the capability influencing factor model is trained using the influencing factors of the capability data of the data provider.

[0113] In one implementation, based on the characteristics of the call volume of AI domain capabilities, the prediction filling model can be an integrated model of LSTM algorithm and call volume influencing factor model, such as a weighted synthesis model of LSTM model and capability influencing factor model.

[0114] Optionally, obtaining the predicted value of the pre-trained prediction filling model for the current capability data of the data provider includes:

[0115] Factors affecting the ability to obtain feedback from the data provider;

[0116] Obtain the predicted value of the current capability data of the data provider based on the predicted capability data and the capability influencing factors by the prediction filling model.

[0117] The call volume influencing factor model can be tailored to the specific characteristics of AI capabilities and business operations, with inputs determined accordingly. The influencing factors and predetermined thresholds can be provided by the data provider based on the characteristics of different capabilities and their daily call volumes. For example, for recommendation and text-based capabilities with daily call volumes in the hundreds of millions, a 10% threshold can be chosen. This allows verification to determine if the acquired capability data exceeds the predicted value by 10%, with influencing factors including the user's business type and the corresponding query time. Machine vision capabilities, however, rarely reach hundreds of millions of calls. If recommendation capabilities are applied to home television content recommendation, peak call volumes will occur during peak television usage times, such as 8 PM to 10 PM; therefore, query time is a key influencing factor. If text-based capabilities are applied to public opinion monitoring, sudden increases will occur when breaking news breaks in society; therefore, news type is a key influencing factor. Therefore, different influencing factors need to be determined based on the characteristics of different capabilities and business scenarios, and different influencing factor models need to be trained accordingly.

[0118] The capability influencing factor model can use Convolutional Neural Networks (CNN) or any other model for regression modeling, and there are no restrictions here.

[0119] Before verification, the prediction filling model needs to be pre-trained. Specifically, the capability data collected by the data provider within the past two years can be divided into training and test sets in a ratio of 7:3, and the model's loss function, number of iterations, network structure, batch size, and other parameters can be determined. The model can then be trained using existing neural network model training methods such as gradient descent, which will not be detailed here.

[0120] After the prediction filling model is trained, it can be used to verify the acquired capability data. Specifically, if the prediction result of the LSTM model is y1 and the prediction result obtained from the capability influencing factor model is y2, then the prediction result of the prediction filling model can be y = a × y1 + (1-a) × y2, where a < 1, and the value of a is an adaptively changing value, which can be obtained through model training.

[0121] Thus, in this implementation, by employing a weighted composite model of the LSTM model and the capability influencing factor model, the accuracy of the prediction filling model can be guaranteed, thereby ensuring the reliability of the verification of the acquired capability data based on this model.

[0122] Optionally, after verifying whether the difference between the second capability data and the predicted value exceeds a predetermined threshold, the method includes:

[0123] If the difference exceeds the predetermined threshold, capability data will be retrieved again from the data provider;

[0124] If the capability data is not reacquired within the preset time, or if the reacquired capability data is empty, the predicted value will be filled with the capability data of the data provider.

[0125] Alternatively, if the reacquired third capability data is the same as the second capability data, then the third capability data is recorded and an alarm message is sent to the data provider.

[0126] In one implementation, if the difference between the second capability data and the predicted value exceeds a predetermined threshold, or if the second capability data is empty, the data acquisition module can be notified to re-acquire capability data, that is, capability data can be re-obtained from the data provider.

[0127] If the capability data collected a second time is the same as the previous data value, that is, if the newly acquired third capability data is the same as the second capability data, then this capability data can be recorded and used to update the prediction filling model. Simultaneously, an alarm message can be triggered to the data provider to remind them to check whether the capability is providing services normally.

[0128] If the data provider fails to return data or returns empty data before the specified time after collecting capability data again, data filling can be initiated. This involves using the predicted values ​​from the prediction filling model to fill the capability data of the data provider. For example, the specified time can be 10:30 AM. If a data point is still incomplete or empty at 10:30 AM, the predicted value from the prediction filling model will be used to fill it in.

[0129] In this way, if the acquired capability data fails verification or there are problems during the data collection and transmission process, the capability data can be made complete by using model prediction values ​​to fill in the capability data, so as to ensure that the final capability data is displayed and improve the user experience.

[0130] Step 104: Display the second capability data that has passed the verification.

[0131] In this step, the verified capability data can be displayed for users to view. Specifically, the capability data can be categorized and statistically processed according to certain rules before being displayed based on the statistical results. For example, the call volume data of all capabilities can be sorted according to the capability provider's capability call volume data, such as by capability provider or by capability type. There are no restrictions on the statistical dimensions here; the statistics can be based on the actual display. Different statistical dimensions can be used for display. If the displayed value is a predicted value, it can be noted on the display page.

[0132] It should be noted that the embodiments of this application may include a data acquisition section, a data processing section, and a data display section. Steps 101 and 102 above constitute the data acquisition section, step 103 the data processing section, and step 104 the data display section. The data acquisition section is executed by a self-query module, a receiving and pushing module, and a parsing and adaptation module. The specific process includes: determining the data acquisition method according to different platforms or devices, parsing the acquired data, and then sending the parsed data to the data processing module. The data processing section is executed by the data processing module, which mainly verifies the capability data acquired by the data acquisition section, and then processes the verified data into the format required by the data display section. When the data display module initiates a real-time query request, if the data cannot be obtained, predicted values ​​are used to fill it in, and then it is processed into a user-friendly format to facilitate users' understanding of the current capability usage. The data display section is executed by the data display module, which is mainly responsible for visually displaying the acquired data.

[0133] In addition, users can also initiate real-time query requests in the data display module. This means that obtaining the first capability data from the data provider can be based on a real-time query request initiated by the user in the data display module. Specifically, if a real-time query request is received on the display end, the request is sent to the data processing module, which then responds to the request. The data processing module first determines the source of the capability data in the request and sends a collection request to the data acquisition module of that data source. If the data acquisition method of the data source is query, the retrieved data is fed back to the data display module. If the data acquisition method of the data source is push, an SMS or email request is sent to the corresponding contact. If no data is found or no push is received within a specified time, the prediction filling model is invoked to return the corresponding predicted value.

[0134] The embodiments of this application have the following advantages: First, different data collection methods are used for multi-source heterogeneous data to meet the data aggregation needs of existing AI capability operation data; Second, an effective prediction-filling model is used to verify the capability data, and when problems occur in the data collection and transmission process, predicted values ​​are used to fill in the gaps to avoid large errors in data display; Third, it can meet the needs of real-time query to a certain extent, and when it cannot meet the needs, predicted values ​​are used to provide feedback to improve the user experience.

[0135] In addition, the aggregation and display of capability data has the following benefits: First, the capability operation data aggregation system enables unified management of capability usage, monitoring and evaluation of capability service quality, continuous accumulation of high-quality and common capabilities from both internal and external sources, further establishing an advanced data-driven production and operation model, and improving user experience. Second, the system aggregates and visualizes capability operation data, facilitating a unified understanding among all parties regarding the usage of capabilities deployed on different devices, providing guidance for product iteration and optimization, and helping to build industry-leading key technologies and capability engines. Third, the system enriches the capability display page, making it easier for users to understand popular capabilities and their corresponding solutions, and stimulating user enthusiasm.

[0136] The data processing method of this application embodiment obtains first capability data from a data provider; converts the first capability data according to a preset data format to obtain converted second capability data; verifies the second capability data; and displays the second capability data that passes the verification. In this way, by automatically collecting capability data from the data provider, the timeliness of data collection can be improved, and by verifying the collected capability data, large errors in the displayed capability data can be avoided, thus meeting the existing needs for the aggregation of capability data in the AI ​​field.

[0137] This application also provides a data processing apparatus. See [link to relevant documentation]. Figure 2 , Figure 2 This is a structural diagram of the data processing apparatus provided in the embodiments of this application. Since the principle by which the data processing apparatus solves the problem is similar to the data processing method in the embodiments of this application, the implementation of this data processing apparatus can refer to the implementation of the method, and repeated details will not be described again.

[0138] like Figure 2 As shown, the data processing device 200 includes:

[0139] The first acquisition module 201 is used to acquire first capability data from the data provider;

[0140] The conversion module 202 is used to convert the first capability data according to a preset data format to obtain the converted second capability data;

[0141] Verification module 203 is used to verify the second capability data;

[0142] The display module 204 is used to display the second capability data that has passed the verification.

[0143] Optionally, the first acquisition module 201 includes:

[0144] The query unit is used to construct a capability query request according to a preset query time and frequency, as well as the pre-stored query interface and request method information of the data provider, when the data provider provides a query interface, and send the capability query request to the query interface of the data provider.

[0145] The first receiving unit is used to receive the first capability data returned by the data provider in response to the query request.

[0146] Optionally, the first acquisition module 201 includes:

[0147] The second receiving unit is used to receive the first capability data pushed by the data provider according to the agreed push time and frequency when the data provider does not provide a query interface.

[0148] Optionally, the first acquisition module 201 further includes:

[0149] The sending unit is configured to send a reminder message to the data provider according to the pre-stored contact information of the data provider when it is detected that the data provider has not pushed the capability data at the agreed push time.

[0150] Optionally, the verification module 203 includes:

[0151] The acquisition unit is used to acquire the predicted value of the pre-trained prediction filling model for the current capability data of the data provider;

[0152] A verification unit is used to verify whether the difference between the second capability data and the predicted value exceeds a predetermined threshold.

[0153] If the difference does not exceed the predetermined threshold, the second capability data verification is deemed successful.

[0154] Optionally, the data processing device 200 further includes:

[0155] The second acquisition module is used to reacquire capability data from the data provider if the difference exceeds the predetermined threshold.

[0156] The filling module is used to fill the predicted value with the capability data of the data provider if the capability data is not reacquired within a preset time, or if the reacquired capability data is empty.

[0157] The recording and sending module is used to record the third capability data and send an alarm message to the data provider if the reacquired third capability data is the same as the second capability data.

[0158] Optionally, the prediction filling model is a weighted composite model of a long short-term memory (LSTM) model and a capability influencing factor model. The LSTM model is trained using the historical capability data of the data provider, and the capability influencing factor model is trained using the influencing factors of the capability data of the data provider.

[0159] Optionally, the acquisition unit is used to acquire the capability influencing factors fed back by the data provider; and to acquire the predicted value of the current capability data of the data provider based on the historical capability data and the capability influencing factors by the prediction filling model.

[0160] The data processing device 200 provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0161] The data processing apparatus 200 of this application embodiment obtains first capability data from a data provider; converts the first capability data according to a preset data format to obtain converted second capability data; verifies the second capability data; and displays the second capability data that passes the verification. In this way, by automatically collecting capability data from the data provider, the timeliness of data collection can be improved, and by verifying the collected capability data, large errors in the displayed capability data can be avoided, thus meeting the aggregation needs of capability data in the existing AI field.

[0162] This application also provides an electronic device. Since the principle by which the electronic device solves the problem is similar to the data processing method in this application, the implementation of this electronic device can be found in the implementation of the method, and repeated details will not be described again. Figure 3 As shown, the electronic device according to an embodiment of this application includes:

[0163] Processor 300 is used to read the program from memory 320 and execute the following procedures:

[0164] Obtain first-hand capability data from data providers;

[0165] The first capability data is converted according to a preset data format to obtain the converted second capability data.

[0166] The second capability data is verified;

[0167] The verified data for the second capability is then displayed.

[0168] Transceiver 310 is used to receive and send data under the control of processor 300.

[0169] Among them, Figure 3 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 300) and memory (memory 320). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface. Transceiver 310 may be multiple elements, including transmitters and transceivers, providing a unit for communicating with various other devices over a transmission medium. Processor 300 is responsible for managing the bus architecture and general processing, and memory 320 may store data used by processor 300 during operation.

[0170] Optionally, the processor 300 is also used to read the program from the memory 320 and perform the following steps:

[0171] When the data provider provides a query interface, a capability query request is constructed according to the preset query time and frequency, as well as the pre-stored query interface and request method information of the data provider, and the capability query request is sent to the query interface of the data provider through the transceiver 310.

[0172] The transceiver 310 receives the first capability data returned by the data provider in response to the query request.

[0173] Optionally, the processor 300 is also used to read the program from the memory 320 and perform the following steps:

[0174] If the data provider does not provide a query interface, the transceiver 310 receives the first capability data pushed by the data provider according to the agreed push time and frequency.

[0175] Optionally, the processor 300 is also used to read the program from the memory 320 and perform the following steps:

[0176] If it is detected that the data provider has failed to push capability data at the agreed push time, a reminder message is sent to the data provider via transceiver 310 according to the pre-stored contact information of the data provider.

[0177] Optionally, the processor 300 is also used to read the program from the memory 320 and perform the following steps:

[0178] Obtain the predicted values ​​of the pre-trained prediction filling model for the current capability data of the data provider;

[0179] Verify whether the difference between the second capability data and the predicted value exceeds a predetermined threshold;

[0180] If the difference does not exceed the predetermined threshold, the second capability data verification is deemed successful.

[0181] Optionally, the processor 300 is also used to read the program from the memory 320 and perform the following steps:

[0182] If the difference exceeds the predetermined threshold, capability data will be retrieved again from the data provider;

[0183] If the capability data is not reacquired within the preset time, or if the reacquired capability data is empty, the predicted value will be filled with the capability data of the data provider.

[0184] Alternatively, if the reacquired third capability data is the same as the second capability data, the third capability data is recorded, and an alarm message is sent to the data provider via transceiver 310.

[0185] Optionally, the prediction filling model is a weighted composite model of a long short-term memory (LSTM) model and a capability influencing factor model. The LSTM model is trained using the historical capability data of the data provider, and the capability influencing factor model is trained using the influencing factors of the capability data of the data provider.

[0186] Optionally, the processor 300 is also used to read the program from the memory 320 and perform the following steps:

[0187] Factors affecting the ability to obtain feedback from the data provider;

[0188] Obtain the predicted value of the current capability data of the data provider based on the predicted capability data and the capability influencing factors by the prediction filling model.

[0189] The electronic device provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0190] Furthermore, the computer-readable storage medium of this application embodiment is used to store a computer program, which can be executed by a processor. Figure 1 The various processes in the method embodiments shown.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0192] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0193] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized by, The method comprises: obtaining first capability data from a data provider; wherein the capability comprises identification, recommendation or text classification, and the first capability data refers to the calling quantity or calling success quantity of the capability; format-converting the first capability data according to a preset data format to obtain second capability data after conversion; verifying the second capability data; displaying the second capability data that passes the verification; the verifying the second capability data comprises: obtaining a predicted value of current capability data of the data provider by a pre-trained prediction filling model; verifying whether a difference between the second capability data and the predicted value exceeds a predetermined threshold; wherein, in the case that the difference does not exceed the predetermined threshold, it is determined that the second capability data passes the verification; the prediction filling model is a weighted combination model of a long short-term memory (LSTM) model and a capability influence factor model, the LSTM model is obtained by training historical capability data of the data provider, and the capability influence factor model is obtained by training influence factors of the capability data of the data provider.

2. The method of claim 1, wherein, The method comprises: in the case that the data provider provides a query interface, constructing a capability query request according to a preset query time and frequency, and a pre-stored query interface and request mode information of the data provider, and sending the capability query request to the query interface of the data provider; receiving first capability data returned by the data provider in response to the query request.

3. The method of claim 1, wherein, The method comprises: in the case that the data provider does not provide a query interface, receiving the first capability data pushed by the data provider according to an agreed pushing time and frequency.

4. The method of claim 3, wherein, The method further comprises: in the case that it is detected that the data provider does not push capability data at the agreed pushing time, sending reminder information to the data provider according to pre-stored contact information of the data provider.

5. The method of claim 1, wherein, After verifying whether the difference between the second capability data and the predicted value exceeds a predetermined threshold, the method comprises: in the case that the difference exceeds the predetermined threshold, re-obtaining capability data from the data provider; if capability data is not re-obtained within a preset time, or the re-obtained capability data is empty, filling the predicted value as the capability data of the data provider; or, if the third capability data re-obtained is the same as the second capability data, recording the third capability data and sending alarm information to the data provider.

6. The method of claim 1, wherein, The method comprises: obtaining capability influence factors fed back by the data provider; obtaining a predicted value of current capability data of the data provider by the prediction filling model according to historical capability data and the capability influence factors.

7. A data processing apparatus, characterized by, The method comprises: The first obtaining module is configured to obtain first capability data from a data provider; wherein the capability comprises identification, recommendation or text classification, and the first capability data refers to a calling quantity or a calling success quantity of the capability; The conversion module is configured to perform format conversion on the first capability data according to a preset data format to obtain second capability data after conversion; The verification module is configured to verify the second capability data; The display module is configured to display the second capability data that passes the verification; The verification module comprises: The obtaining unit is configured to obtain a predicted value of a pre-trained prediction filling model for current capability data of the data provider; The verification unit is configured to verify whether a difference between the second capability data and the predicted value exceeds a predetermined threshold value; In a case where the difference does not exceed the predetermined threshold value, it is determined that the second capability data passes the verification. The prediction filling model is a weighted combination model of a long short-term memory (LSTM) model and a capability influence factor model, the LSTM model is obtained by training historical capability data of the data provider, and the capability influence factor model is obtained by training influence factors of the capability data of the data provider.

8. An electronic device comprising: The transceiver, the memory, the processor and the computer program stored in the memory and executable on the processor are characterized in that the processor is configured to read the program in the memory to implement the steps in the data processing method according to any one of claims 1 to 6.

9. A computer readable storage medium for storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the data processing method according to any one of claims 1 to 6.

Images