Data value assessment method and electronic device
The data value assessment method addresses inefficiencies in data lake construction by calculating field values based on access frequencies and business weights, accurately reflecting data importance and optimizing storage through association relationship analysis.
Patent Information
- Application Number
- CN202410057985.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-11-30
AI Technical Summary
Current data lake construction methods fail to accurately assess data value, leading to inefficient storage and resource utilization due to low model reuse, misjudgment of data importance, and inability to handle mixed database scenarios.
A data value assessment method that calculates field values in a data lake by considering access frequencies and business weights, and determines association relationships through parsing and analyzing code warehouses to reflect data value accurately.
Enhances data value assessment accuracy, optimizing storage utilization by identifying the true value of data fields and tables, reducing waste and improving resource allocation.
Smart Images

Figure CN117891852B_ABST
Abstract
Description
[0001] This disclosure is a divisional application. The application number of the original application is 202211529143.2, and the original application date is November 30, 2022. The entire content of the original application is incorporated into this application by reference. Technical Field
[0002] The present application relates to the field of data processing technology, and in particular to a data value assessment method and electronic equipment. Background Art
[0003] In the construction of a data lake, data developers will integrate a large amount of source system data from the transaction system, and through a series of data calculations, build application scenarios such as reports and data services. In the construction scenario, the following problems often occur: the model reuse of the data lake construction is low, or the fields have not been used for a long time, resulting in a large amount of storage space waste.
[0004] The current solution is to analyze the execution plan or access log of a single data warehouse (such as Hive) to identify tables with large or small access times, so as to evaluate the value of data. However, the above solution does not give business meaning to the data, and it is impossible to intuitively judge the value of the data, which may lead to misjudgment of the data value. Summary of the invention
[0005] The embodiments of the present application provide a data value assessment method and an electronic device, which can more accurately reflect the value of data.
[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:
[0007] In a first aspect, a data value assessment method is provided for assessing the field value of each field in a data lake, wherein the data lake includes multiple fields, and there is an association relationship between the multiple fields, and the association relationship is used to indicate the source of a target field facing a user layer in the data lake. The method includes: obtaining the access frequency of multiple target fields in the data lake; respectively calculating the product of a business weight corresponding to each target field and the access frequency of the target field to obtain the field value of the target field; wherein the business weight indicates the importance of the target field in the corresponding business scenario; and calculating the field value of other fields in the data lake except the target field based on the field value corresponding to each target field in the multiple target fields and the association relationship.
[0008] In an embodiment of the present application, business personnel can be supported to configure business weights for each target field, add the influence of the business weight when calculating the field value, and pass the business weight to the associated node during value assessment, thereby more accurately reflecting the data value. In addition, after obtaining the data value of a single field, this solution can obtain all direct associations and transitive associations of the field by parsing and analyzing the code warehouse. According to the above-mentioned associations, the field value can be passed to its source field, so that the value of its source field can be obtained. The value of the field on the processing link can be more accurately reflected.
[0009] In a possible implementation of the first aspect, the data lake includes multiple data tables, each of which includes multiple fields; after calculating the field value of other fields in the data lake except the target field based on the field value and association relationship corresponding to each target field in the multiple target fields, the above method also includes: counting the sum of the field values of all fields included in each data table in the multiple data tables, and the number of fields; respectively calculating the ratio of the sum of the field values of each data table to the number of fields to obtain the table value of the data table. The embodiment of the present application can more accurately obtain the table value of the data table based on the field value with the business weight added.
[0010] In a possible implementation of the first aspect, before calculating the field values of other fields in the data lake except the target field based on the field value and the association relationship corresponding to each target field in the multiple target fields, the method further includes: determining the association relationship between the multiple fields in the data lake. The embodiment of the present application can determine the association relationship between the multiple fields in the data lake by parsing and analyzing the code warehouse.
[0011] In a possible implementation manner of the first aspect, the determining the association relationship between multiple fields in the data lake includes: obtaining a database script from a code warehouse, and determining a SQL script to be converted from the database script based on a target field; performing lexical parsing and grammatical parsing on the SQL script to be converted based on written lexical and grammatical parsing rules to obtain a grammatical parsing tree; and traversing the grammatical parsing tree to obtain the association relationship between the fields in the SQL script to be converted.
[0012] In a possible implementation of the first aspect, the pre-order traversal of the syntax parse tree to obtain the association relationship between the fields in the SQL script to be converted includes: traversing the syntax parse tree to determine the leaf nodes in the syntax parse tree; and parsing starting from the leaf nodes based on the visitor pattern of the syntax parse tree to obtain the association relationship between the fields in the SQL script to be converted.
[0013] In a possible implementation of the first aspect, the association relationship includes a direct association relationship and a transitive association relationship between the target field and other fields in the data lake; wherein the field that has a direct association relationship with the target field is a direct source for generating the target field, and the field that has a transitive association relationship with the target field is an indirect source for generating the target field. The above-mentioned calculation of the field value of other fields in the data lake except the target field based on the field value and association relationship corresponding to each target field in the multiple target fields includes: for each other field in the data lake except the target field, calculating the sum of the field values of one or more target fields that have a direct association relationship or a transitive association relationship with other fields as the field value of other fields.
[0014] In a possible implementation manner of the first aspect, the obtaining access frequencies of the multiple target fields in the data lake includes: obtaining access records of the multiple target fields in the data lake, and obtaining an access frequency of each of the multiple target fields according to the access records.
[0015] In a possible implementation of the first aspect, the access record is report data; the access frequency of each target field in the multiple target fields is obtained based on the access record, including: obtaining the access log of the reporting tool, parsing the access log, and obtaining the access frequency of each target field in the multiple target fields.
[0016] In a possible implementation of the first aspect, the access record is data in a data service platform; and obtaining the access frequency of each target field in the multiple target fields according to the access record includes: calling the data service log and parsing the data service log to obtain the access frequency of each target field in the multiple target fields. The access record may also be data in other application scenarios, and the report data and the data in the data service platform are only examples.
[0017] In a second aspect, an electronic device is provided, comprising: a communication module, a display screen, a memory and one or more processors; the communication module, the display screen, the memory and the processor are coupled; the memory is used to store computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the electronic device, the electronic device executes the above-mentioned data value assessment method.
[0018] In a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on a computer, the computer can execute the data value assessment method described in any one of the first aspects.
[0019] In a fourth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the data value assessment method described in any one of the first aspects.
[0020] In a fifth aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting a first device to implement the functions involved in the first aspect. In one possible design, the device also includes a memory for storing program instructions and data necessary for the first device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.
[0021] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A schematic diagram of user access in different business scenarios provided in an embodiment of the present application;
[0023] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;
[0024] Figure 3 A schematic diagram of an association relationship between target fields provided in an embodiment of the present application;
[0025] Figure 4 A schematic diagram of a grammar parsing tree provided in an embodiment of the present application;
[0026] Figure 5 A schematic diagram of a location record of a field value provided in an embodiment of the present application;
[0027] Figure 6 A schematic diagram of an association relationship between fields provided in an embodiment of the present application;
[0028] Figure 7 A schematic diagram of an integrated solution for data value assessment provided in an embodiment of the present application;
[0029] Figure 8 A flowchart of a data value calculation solution provided in an embodiment of the present application. DETAILED DESCRIPTION
[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0031] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.
[0032] In the construction of data lakes, data developers will integrate a large amount of source system data from transaction systems, and through a series of data calculations, build application scenarios such as reports and data services. The following problems may arise in the construction of application scenarios: the model reuse of data lake construction is low, or the fields have not been used for a long time, resulting in a large amount of storage space waste; some data services are used less frequently, but the processing logic is complex, occupying a large amount of storage and computing resources; only the high and low frequencies of directly accessed reports can be counted, and it is difficult to estimate the value of the model for the original data under the report processing link; the current data middle platform does not have a good quantitative indicator reference for the evaluation of the quality of the model in the data lake.
[0033] To solve the above problems, the current solution is to analyze the execution plan or access log of a single data warehouse (such as Hive) to identify tables with larger or smaller access times, so as to evaluate the data value.
[0034] However, when targeting offline computing scenarios, the above method identifies that the number of times key data models are accessed is only related to the frequency of offline scheduling, which leads to misjudgment of data value. Business meaning is not given to the data. In some application scenarios, the value of the data cannot be intuitively judged simply by looking at the popularity of the data. In a frequently accessed table, the value of the field cannot be accurately judged. For example, only a few fields in the table are actually frequently used, and most fields are low-value data, which leads to consumption of storage space. In addition, multiple databases are often mixed in big data scenarios, and the above method can only identify a single type of data warehouse.
[0035] Among them, the hotness and coldness of data refers to the classification of data according to the frequency of data access. Data can be divided into hot data, warm data and cold data. Hot data, for example, can be data within half a year, and users often query them. Correspondingly, hot data is placed in hot storage. For example, popular videos have a large number of visits and high hardware requirements. They often have high-performance servers, storage, and bandwidth support. Warm data, for example, can be data from half a year to a year. Users sometimes need to query data within a year online. These data are also suitable for storage in the database. If the amount of data is not large, warm data and hot data can be ignored. Cold data is data that is not used very much. This data can be stored in a file system with lower performance and lower price. Correspondingly, cold data is placed in cold storage. For example, unpopular videos have a small number of visits and do not require high performance from the data center. They are generally allocated to servers with lower performance.
[0036] See also Figure 1 Scenario 1: In this business scenario, users query reports 100 times a day, and Table A and Table B are scheduled once a day. If the traditional hot and cold data identification method is used, it can be obtained that the user access volume of the report is 100 times a day, which is marked as a hot table; Table A and Table B are only called once a day, which are marked as cold tables. If the data value is judged based on the identified hot and cold tables, the value of Table A and Table B is low. In fact, due to the large number of user accesses to the report, the value of the report is high, and the report is obtained from Table A and Table B, so the value of Table A and Table B should also be high. Therefore, judging the data value based on the identified hot and cold tables may lead to misjudgment of the data value.
[0037] See also Figure 1 In scenario 2, in this business scenario, the user queries the report once a day, and Table A and Table B are scheduled 120 times a day. If the traditional cold and hot data identification method is used, it can be obtained that the user access to the report is 1 time a day, which is marked as a cold table; Table A and Table B are called 120 times a day, which are marked as hot tables. If the data value is judged based on the identified cold and hot tables, the value of Table A and Table B is higher. In fact, since the user access to the report is small, the value of the report is low, and the report is obtained from Table A and Table B, then the value of Table A and Table B should also be low. Therefore, judging the data value based on the identified cold and hot tables may lead to misjudgment of the data value.
[0038] See also Figure 1 In scenario 3, users access two types of databases. Therefore, different data stacks need to be customized for different databases to analyze the access volume, while traditional solutions can only identify the number of accesses for a single data warehouse type.
[0039] In order to solve the above problems, the embodiment of the present application provides a data value assessment method, which is applied to a data lake to assess the field value of each field in the data lake, and thus can more accurately assess the value of the table in the data lake. The data lake includes multiple fields, and there is an association relationship between the multiple fields. The association relationship is used to indicate the source of the target field facing the user layer in the data lake.
[0040] The following will describe the implementation of the present application in detail with reference to the accompanying drawings. A data value assessment method provided by the present application embodiment can be applied to electronic devices. Figure 2 As shown, the electronic device 200 may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.
[0041] Among them, the above-mentioned sensor module 280 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.
[0042] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0043] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0044] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0045] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.
[0046] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0047] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments, the electronic device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0048] The charging management module 240 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 may receive charging input from a wired charger through the USB interface 230. In some wireless charging embodiments, the charging management module 240 may receive wireless charging input through a wireless charging coil of the electronic device 200. While the charging management module 240 is charging the battery 242, it may also power the electronic device through the power management module 241.
[0049] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the external memory, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 241 can also be set in the processor 210. In other embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.
[0050] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.
[0051] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0052] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 200. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation.
[0053] The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the same device as at least some modules of the processor 210.
[0054] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 270A, a receiver 270B, etc.), or displays an image or video through a display screen 294. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 210 and be set in the same device as the mobile communication module 250 or other functional modules.
[0055] The wireless communication module 260 can provide wireless communication solutions for application in the electronic device 300, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.
[0056] The wireless communication module 260 may be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210. The wireless communication module 260 may also receive signals to be sent from the processor 210, modulate the frequencies of the signals, amplify the signals, and convert the signals into electromagnetic waves for radiation via the antenna 2.
[0057] In some embodiments, the antenna 1 of the electronic device 200 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the electronic device 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).
[0058] The electronic device 200 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.
[0059] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc.
[0060] The electronic device 200 can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.
[0061] The ISP is used to process the data fed back by the camera 293. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 293.
[0062] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 200 may include 1 or N cameras 293, where N is a positive integer greater than 1.
[0063] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0064] Video codecs are used to compress or decompress digital videos. The electronic device 200 may support one or more video codecs. Thus, the electronic device 200 may play or record videos in a variety of coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0065] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0066] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0067] The internal memory 221 may be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 may execute the instructions stored in the internal memory 221, and the internal memory 221 may include a program storage area and a data storage area.
[0068] The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0069] The electronic device 200 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.
[0070] The touch sensor is also called a "touch panel". The touch sensor can be arranged on the display screen 294, and the touch sensor and the display screen 294 form a touch screen, also called a "touch screen". The touch sensor is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 294. In other embodiments, the touch sensor can also be arranged on the surface of the electronic device 200, which is different from the position of the display screen 294.
[0071] In the embodiment of the present application, the electronic device 200 can detect the touch operation input by the user on the touch screen through the touch sensor, and collect one or more of the touch position, touch area, touch direction, and touch time of the touch operation on the touch screen. In some embodiments, the electronic device 200 can determine the touch position of the touch operation on the touch screen by combining the touch sensor and the pressure sensor.
[0072] The key 290 includes a power key, a volume key, etc. The key 290 can be a mechanical key or a touch key. The electronic device 200 can receive key input and generate key signal input related to user settings and function control of the electronic device 200.
[0073] Motor 291 can generate vibration prompts. Motor 291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 294, motor 291 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0074] The indicator 292 may be an indicator light, which may be used to indicate the charging status, power change, message, missed call, notification, etc. The SIM card interface 295 is used to connect the SIM card. The SIM card may be connected to or disconnected from the electronic device 200 by inserting the SIM card interface 295 or pulling the SIM card interface 295 out. The electronic device 200 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 may support Nano SIM card, Micro SIM card, SIM card, etc.
[0075] The methods in the following embodiments can all be implemented in the electronic device 200 having the above hardware structure.
[0076] The data value assessment method provided in the embodiment of the present application may mainly include four modules: a user data analysis module, a value weight configuration module, an association relationship analysis module and a value assessment module.
[0077] In an embodiment of the present application, the user data parsing module is used to obtain the access frequency of multiple target fields facing the user layer in the data lake. Specifically, the access record of multiple target fields in the data lake is obtained, and the access frequency of each target field in the multiple target fields is obtained according to the access record. Among them, the access frequency of the target field is the number of times the target field is accessed per unit time.
[0078] Among them, we can obtain the access records of user data in the data lake. User data includes multiple target fields for the user layer. For users, the main data they come into contact with on the Internet is the top-level data, such as videos, files, and games, which is network application data. Therefore, user data is also the top-level data.
[0079] If in the report application scenario, that is, the access record is report data, the access log of the self-service report tool can be obtained and parsed to obtain the access frequency of each target field in the multiple target fields. If in the data service application scenario, that is, the access record is the data in the data service platform, the data service log is called and parsed to obtain the access frequency of each target field in the multiple target fields. The above-mentioned report data and the data in the data service platform are examples of embodiments of the present application, and the user data can also be data in other application scenarios.
[0080] In an embodiment of the present application, the value weight configuration module is used to configure the value weight of the target field that the user directly accesses. That is, business weights are configured for multiple target fields in the data lake, where the business weight is used to indicate the importance of the field in the corresponding business scenario. Business personnel can configure the weights of the corresponding target fields according to different indicators. For example, in business scenario 1, target field 1 corresponding to indicator 1 is more important than target field 2 corresponding to indicator 2, then the weight value corresponding to target field 1 is greater than the weight value corresponding to target field 2.
[0081] The business weight configuration of the target field can be obtained in real time when the field value is evaluated. Alternatively, the business weight configuration of the target field can be configured periodically. For example, in different business scenarios, the importance of indicators may change after a period of time. At this time, the business weight configuration of the target field can be reconfigured.
[0082] In an embodiment of the present application, the association relationship parsing module is used to determine the association relationship between multiple fields in the data lake. The association relationship includes a direct association relationship and a transitive association relationship between a target field and other fields in the data lake. A field that has a direct association relationship with a target field is a direct source for generating the target field, and a field that has a transitive association relationship with a target field is an indirect source for generating the target field.
[0083] For example, if target field 1 is directly associated with field 2, it means field 2 generates target field 1. There can be one or more fields that are directly associated with the target field. Figure 3 , Figure 3 A schematic diagram showing the association relationship between target fields. Figure 3 As shown, the user access record is stored in the form of a data table, and the user access record includes Table 1 and Table 2. Table 1 includes field 11, and Table 2 includes field 21. The direct source of Table 1 and Table 2 is Table 3, and Table 3 includes field 31, field 32, and field 33. Figure 3 The arrows between the fields in Table 3 and the fields in Tables 1 and 2 indicate the direct association relationship between the target fields (fields 11 and 21). Field 31 points to field 11, that is, field 31 generates field 11, and field 31 is the direct source of field 11. Field 31, field 32, and field 33 point to field 21, that is, field 31, field 32, and field 33 generate field 21, and field 31, field 32, and field 33 are all direct sources of field 11.
[0084] Figure 3 It also shows the transitive association between the target field and other fields in the data lake. Figure 3 As shown, the next level table of Table 1 and Table 2 is Table 3, and the direct sources of the fields contained in Table 1 and Table 2 are the fields contained in Table 3. The next level table of Table 3 is Table 4, Table 5 and Table 6, and the next level tables of Table 4, Table 5 and Table 6 are Table 7, Table 8 and Table 9; the fields contained in Table 4, Table 5, Table 6, Table 7, Table 8 and Table 9 are all indirect sources of the fields contained in Table 1 and Table 2. Figure 3 There are 4 layers of fields in the table, with the data layer where the fields in Table 1 and Table 2 are located as Layer 0. Table 1 includes Field 11, and Table 2 includes Field 21. Then, Layer 1 includes Table 3, which includes Field 31, Field 32, and Field 33. Layer 2 includes Table 4, Table 5, and Table 6. Table 4 includes Field 41, Table 5 includes Field 51, and Table 6 includes Field 61. Layer 3 includes Table 7, Table 8, and Table 9. Table 7 includes Field 71, Table 8 includes Field 81 and Field 82, and Table 9 includes Field 91 and Field 92. Figure 3The arrows point to the relationship between the fields. Field 71 generates field 41, field 41 generates field 31, field 31 generates field 11 and field 21; field 81 generates field 51, field 51 generates field 32 and field 33, field 32 and field 33 both generate field 21; field 91 generates field 61, field 61 generates field 33, field 33 generates field 21.
[0085] In an embodiment of the present application, a syntax tree can be obtained by parsing a database script. Specifically, a database script is obtained from a code warehouse, and a SQL script to be converted is determined from the database script based on a target field. By writing lexical syntax tree parsing rules and using syntax parsing methods, such as using an open source syntax parsing tool, the SQL script to be converted can be parsed into a tree structure for subsequent traversal. Similarly, the platform metadata can also be parsed to obtain a syntax tree.
[0086] For example, the obtained SQL script is as follows:
[0087] insert overwrite table target.table1
[0088] select a.user_id as user_id,a.activity_id as activity_id
[0089] from source.table2 a
[0090] The above SQL script can be parsed into a syntax tree through self-developed syntax parsing rules. Figure 4 A portion of the grammar parse tree of the SQL script is shown. Then, the association relationship between the fields in the SQL script can be obtained by traversing the grammar tree and the visitor pattern.
[0091] For example, traversing the syntax parse tree, the insertion syntax is first found, and the visitor of the insertion syntax is entered. The table name subtree under the node is obtained as the downstream table name of the association relationship, that is, Figure 4 The target table in the query is named target.table1. Next, the subquery syntax is traversed to find whether there is a from table syntax under the syntax tree. If so, the source table name and alias are recorded. Figure 4The source table in the query is named source.table2 and its alias is a. If from exists but table does not exist, it means it is a nested subquery. The node id of the from subquery is established to establish a parent-child relationship with the node id of this node, which is convenient for subsequent analysis. Then, the select list is traversed to obtain all selectitems under this select list. The select list alias is used as the target field and the prefix is used as the source field. Figure 4 The target fields in are: userid, activity id. Finally, get the alias of the source field. Field name, convert the alias a to the table name. And record the source field, Figure 4 The source fields in are: source.table2.user id, source.table2.activity id. If expr is a function, only the variable content involved in the function is obtained as the source field. Finally, expr is recorded as a conversion rule for easy backtracking.
[0092] Among them, the above method of obtaining the association relationship between each field by parsing the syntax tree and traversing Antlr4 is only an example, and other related technologies can also be used to achieve the same effect, and this application does not limit this.
[0093] Thus, through Figure 4 The grammar parse tree and the visitors corresponding to the grammars of different nodes can determine the association between the fields of the SQL script in the above example.
[0094] The embodiment of the present application does not limit the above-mentioned traversal method, which can be a pre-order traversal, an in-order traversal, or a subsequent traversal.
[0095] In the embodiment of the present application, different visitor methods can be used to perform business logic processing on leaf nodes of different syntax trees.
[0096] In the embodiment of the present application, the value assessment module is used to combine the user data analysis module, the value weight configuration module and the association relationship analysis module to calculate the true value of the field.
[0097] First, the field value of the target field is obtained by calculating the product of the business weight corresponding to each target field and the access frequency of the target field. For example, the number of visits per unit time of field A1 is 20, and the business weight configured by the user is 1.5, then the value of field A1 is 1.5*20=30. The number of visits per unit time of field B1 is 50, and the business weight configured by the user is 0.8, then the value of field A1 is 50*0.8=40. Figure 5 As shown, table A includes field A1, and table B includes field B1. After the field values are calculated, the corresponding field values can be recorded in table A and table B. Figure 5 The recording locations shown in are only examples and are not limited in this application.
[0098] Then, based on the field value corresponding to each target field in the multiple target fields and the association relationship between the multiple fields, the field values of other fields in the data lake except the target field are calculated.
[0099] Among them, the association relationship analysis module can determine the association relationship between multiple fields in the data lake. Then, for each field in the data lake except the target field, the sum of the field values of one or more target fields that have a direct association relationship or a transitive association relationship with other fields is calculated as the field value of other fields.
[0100] For example, see Figure 6 ,based on Figure 6 From the relationship between the fields shown, we can conclude that field C1 generates field A1 and field B1, and field C1 has a direct relationship with field A1 and field B1, so the value of field C1 is accumulated by A1 and B1, and the value of field C1 is 70. Field D1 generates field C1, so field D1 is the indirect source of field A1 and field B1, and field D1 has a transitive relationship with field A1 and field B1, so the value of field D1 is accumulated by A1 and B1, and the value of field D1 is 70. By analogy, we can conclude that the value of field C2 is 40, the value of field C3 is 40, the value of field E1 is 80, the value of field F1 is 80, the value of field G1 is 70, the value of field H1 is 80, and the value of field I1 is 40. Since field H2 and field I2 are not direct or indirect sources of field A1 and field B1, the values of field H2 and field I2 are 0, which means that field H2 and field I2 will not be used when users access them and are low-value fields.
[0101] The embodiment of the present application transfers the field value to its source field according to the generation logic, thereby obtaining the value of its source field, and the value assessment of each source field is more accurate.
[0102] In the embodiment of the present application, after obtaining the field values of all fields in the data lake, the value of the data table in the data lake can be calculated. Specifically, the sum of the field values of all fields included in each data table in multiple data tables and the number of fields are counted, and the ratio of the sum of the field values of each data table to the number of fields is calculated to obtain the table value of the data table. For example, Figure 6 As shown, table H contains fields H1 and H2. The total value of the fields in table H is 80, and the number of fields is 2, so the value of table H is 40. Among them, table 1 is Figure 6 The value of each data table in .
[0103] Table 1
[0104] surface A B C D E F G H I value 30 40 50 70 80 40 70 40 20
[0105] The data evaluation method provided in the embodiment of the present application can support business personnel to configure business weights for each target field, add the influence of the business weight when calculating the field value, and pass the business weight to the associated node during value evaluation, thereby more accurately reflecting the data value. In addition, after obtaining the data value of a single field, this solution can obtain all direct associations and transitive associations of the field by parsing and analyzing the code warehouse. According to the above-mentioned associations, the field value can be passed to its source field, so that the value of its source field can be obtained. The defects of offline scheduling key model misjudgment, view value misjudgment, and cross-library access misjudgment in traditional solutions can be solved.
[0106] See also Figure 7 , Figure 7 A schematic diagram of an integrated solution for data value assessment provided for an embodiment of the present application. Among them, the user data analysis module is connected to the data service & data display module. The data in the data service & data display module is facing the user layer, which can be report data and data in the data service platform. If the access record is report data, the user data analysis module parses the access log of the self-service report tool to obtain the access frequency of the target field of the access report. If the access record is data in the data service platform, the user data analysis module parses the data service log of the data service platform to obtain the access frequency of the target field of the access data service.
[0107] Then, the value weight configuration module configures the business weight of the target field accessed by the user. The access frequency of the target field is combined with the business weight to obtain the true value of the field directly accessed by the user. Next, the database script and platform metadata are parsed by the association relationship parsing module to obtain the field tree of the association relationship between the fields. Among them, the database script is obtained from the code warehouse, and the platform metadata is obtained from the big data platform tool. The big data platform tools include the data integration platform and the data acquisition platform. Among them, the database script contains one or more groups of SQL statements. In other words, the association relationship parsing module can parse the direct SQL code or log obtained from the database script, platform metadata, and execution log.
[0108] If the association relationship parsing module obtains logs, the log parsing module of the corresponding system is used to parse the running logs into the association relationship between fields and fields, tables and tables, and tables and fields. If the association relationship parsing module obtains SQL scripts, the SQL scripts are first parsed into syntax trees, and then the field tree of the association relationship between fields is obtained by traversing the syntax tree and the visitor pattern.
[0109] Finally, by combining the real value of the target field with the above field tree, the value of all fields in the data lake can be recursively calculated. And based on the value of each field, the value of each data table in the data lake can be calculated.
[0110] See also Figure 8 , Figure 8 The following is a flow chart of a data value calculation solution provided in an embodiment of the present application. Figure 8 As shown, the data service log of the user accessing the data service and the access SQL of the user accessing the report are collected. The data service log is parsed by the log parser to count the number of times the target field is accessed per unit time; the access SQL is parsed by the SQL parser, that is, the SQL syntax parsing tree system described above is used to count the number of times the target field is accessed per unit time. Then, the field value of the target field can be obtained based on the product of the number of times the target field is accessed per unit time and the business weight of the target field configured by the user. The specific formula is as follows:
[0111] 0-layer data value: value(0) = visit*ratio;
[0112]
[0113] Among them, the 0-layer data value refers to the field value of the target field visited by the user; visit indicates the number of visits to the target field per unit time, and ratio indicates the business weight of the target field. The k-layer data value refers to the field value of the field that has a direct association with the target field (k=1), and the field value of the field that has a transitive association with the target field (k>1). According to the above formula, the k-layer data value is the sum of the field values of one or more target fields that have a direct association or transitive association with the k-layer field.
[0114] Finally, according to the field value of each field, the total field value of all fields in each data table in multiple data tables and the number of fields are counted, and the ratio of the total field value to the number of fields in each data table is calculated to obtain the table value of the data table. The specific formula is as follows:
[0115]
[0116] The data value assessment method provided in this embodiment is used in a data lake, and can evaluate the quantitative indicators of the model constructed in the data lake to guide data governance behaviors such as model optimization and storage cost optimization.
[0117] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected by lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.
[0118] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed in the above-mentioned method embodiment.
[0119] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed in the above method embodiment.
[0120] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0121] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0122] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0123] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0125] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims
1. A data value assessment method, characterized in that: For evaluating the field value of each field in a data lake, the data lake includes a plurality of fields, there is an association relationship between the plurality of fields, the association relationship is used to indicate the source of a target field facing a user layer in the data lake, the method comprising: Obtain access frequencies of multiple target fields in the data lake; The product of the business weight corresponding to each target field and the access frequency of the target field is calculated respectively to obtain the field value of the target field; wherein the business weight indicates the importance of the target field in the corresponding business scenario; the business weight is configured according to the indicator corresponding to the target field; the business weight is configured in real time or periodically; Based on the field value corresponding to each target field in the multiple target fields and the association relationship, the field value of other fields in the data lake except the target field is calculated; wherein the field value of the field that has no association relationship with the target field is determined as a preset value; The association relationship includes a direct association relationship and a transitive association relationship between the target field and other fields in the data lake; wherein the field having the direct association relationship with the target field is a direct source for generating the target field, and the field having the transitive association relationship with the target field is an indirect source for generating the target field; for each other field in the data lake except the target field, the sum of the field values of one or more target fields having the direct association relationship or the transitive association relationship with the other field is calculated as the field value of the other field.
2. The data value assessment method according to claim 1, characterized in that: The data lake includes multiple data tables, each of which includes multiple fields; After calculating the field values of other fields in the data lake except the target field based on the field value corresponding to each target field in the multiple target fields and the association relationship, the method further includes: Counting the total field values and the number of fields of all fields included in each data table of the multiple data tables; The ratio of the sum of the field values of each data table to the number of fields is calculated respectively to obtain the table value of the data table.
3. The data value assessment method according to claim 1 or 2, characterized in that: Before calculating the field values of other fields in the data lake except the target field based on the field value corresponding to each target field in the multiple target fields and the association relationship, the method further includes: Determine the association relationship between the multiple fields in the data lake.
4. The data value assessment method according to claim 3, characterized in that: The determining the association relationship between the multiple fields in the data lake includes: Obtain a database script from the code warehouse, and determine a SQL script to be converted from the database script based on the target field; Based on the written lexical and grammatical parsing rules, the SQL script to be converted is subjected to lexical parsing and grammatical parsing to obtain a grammatical parsing tree; The syntax parse tree is traversed to obtain the association relationship between the fields in the SQL script to be converted.
5. The data value assessment method according to any one of claims 1, 2, and 4, characterized in that: The obtaining access frequencies of multiple target fields in the data lake includes: Access records of multiple target fields in the data lake are obtained, and access frequency of each target field in the multiple target fields is obtained according to the access records.
6. The data value assessment method according to claim 5, characterized in that: The access record is report data; obtaining the access frequency of each target field in the plurality of target fields according to the access record includes: The access log of the reporting tool is obtained and the access log is parsed to obtain the access frequency of each target field in the multiple target fields.
7. The data value assessment method according to claim 5, characterized in that: The access record is data in the data service platform; obtaining the access frequency of each target field in the multiple target fields according to the access record includes: The data service log is called and parsed to obtain the access frequency of each target field in the multiple target fields.
8. An electronic device, characterized in that: The electronic device comprises: a communication module, a display screen, a memory and one or more processors; the communication module, the display screen, the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code comprises computer instructions, and when the computer instructions are executed by the electronic device, the electronic device executes the method as described in any one of claims 1-7.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed in the mobile terminal, the mobile terminal executes the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Data table evaluation method and device
CN112817834A
Metadata scoring method and system for business object, and storage medium
CN115114273A
Data report generation method and device, electronic equipment and storage medium
CN115345139A
