Online data acquisition and display method and device and electronic equipment

Abstract

The invention discloses an online data acquisition and display method and device and electronic equipment, and relates to the technical field of online data acquisition, and the method comprises the steps: inputting a data acquisition demand, calculating a comprehensive similarity coefficient of current input and historical data, and determining an acquisition data source of the online data; the method comprises the following steps: acquiring initial data, respectively extracting features of the same data type in an initial data set, dividing main features and additional features, dividing data containing all the main features into an intermediate data set, calculating an additional coefficient of each piece of data, and dividing an undetermined data set; and calculating a value coefficient of each piece of data, calculating a typical index of each piece of data, and determining a display priority of each piece of data under each data type. According to the method, the importance and representativeness of each piece of data under a specific data type can be comprehensively evaluated, and the most valuable, most representative and frestest data can be preferentially displayed under limited display resources.

Description

Technical Field

[0001] The present invention relates to the technical field of online data acquisition, and in particular to an online data acquisition and display method and device and electronic equipment. Background Art

[0002] In today's digital age, data has become the core driving force for corporate decision-making and operations. With the popularization of the Internet and the booming development of e-commerce, the acquisition and display of online data is particularly important. These data can not only help companies gain insight into market trends and understand consumer behavior, but also provide strong support for product recommendations, marketing strategies, etc. With the rapid development of big data and artificial intelligence technologies, online data acquisition and display methods have emerged. These methods use automated means to capture, process and display data from Internet platforms, greatly improving the efficiency and quality of data acquisition.

[0003] In the Chinese invention application with application publication number CN110363600A, a method and device for acquiring and displaying online data based on offline goods and an electronic device are disclosed, including acquiring an online address corresponding to the product identification of the offline goods; accessing the online address and reading the online data corresponding to the offline goods. The storage address of the online data associated with the offline goods is acquired through the product identification of the offline goods, and then the online data stored at the address is read, the offline goods are closely associated with the online data, and the online resources are fully utilized to provide decision-making supplements for users to make offline shopping.

[0004] In the above invention application, the storage address of the online data associated with the offline product is obtained through the product identification of the offline product, and then the online data stored at the address is read to closely associate the offline product with the online data. However, the online data is often huge in volume, and a large amount of unfiltered and unsorted data is presented to the user at the same time, which can easily cause information overload. It is difficult for users to quickly find key information, and they may even ignore the content that is important to them. This information overload not only wastes the user's time, but also may affect their decision-making and judgment ability. And for systems that need to process large amounts of data in real time, if the data is not filtered and sorted, the system may need to load and render more data, thereby increasing the burden and response time of the server.

[0005] To this end, the present invention provides an online data acquisition and display method and device and an electronic device. Summary of the invention

[0006] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides an online data acquisition and display method, device and electronic device. The present invention first screens the data source, concentrates on the high-value data source, avoids the repeated search and verification of the new data source, and then gradually screens the data, which can significantly reduce the amount of data to be processed later and achieve refined data screening. It not only retains the data that may have display value, but also avoids including all data in the subsequent processing, thereby improving the processing efficiency. Finally, according to the additional coefficient of each data and value coefficient Calculate the typical index of each data in the pending data set , determine the display priority of each data under each data type, and comprehensively evaluate the importance and representativeness of each data under a specific data type, ensuring that the most valuable, representative and fresh data are displayed first under limited display resources, thereby solving the technical problems recorded in the background technology.

[0007] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: an online data acquisition and display method, comprising the following steps: Input data acquisition requirements, based on the similarity of acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data , determine the data source for obtaining online data; Obtain initial data from the online data acquisition data source, use a variety of different feature extraction models to extract features of the same data type in the initial data set, divide the main features and additional features, divide the data containing all the main features into the intermediate data set, and calculate the additional coefficient of each data in the intermediate data set Fj , the additional coefficient of each data in the intermediate data set Fj Greater than The data is divided into the pending data set; Get the number of views for each data in the pending data set Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz , based on the additional coefficients for each data and value coefficient Calculate the typical index of each data in the pending data set , determine the display priority of each data under each data type.

[0008] Further, based on the similarity of the acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data :

[0009] in, i Indicates the time sequence number of all historical data acquisition, .

[0010] Furthermore, the comprehensive similarity coefficient Sort from large to small, extract the data sources corresponding to the top 10 historical data, count the number of occurrences of the data sources and sort from large to small, extract the top 3 data sources as the data sources for this online data acquisition.

[0011] Furthermore, a depth-first search algorithm was used to obtain initial data in this online data acquisition data source based on the required data type, range, time span, and subject elements. After preprocessing the initial data to remove noise, outliers, duplicate values, and missing values ​​in the data, an initial data set was constructed.

[0012] Furthermore, a variety of different feature extraction models are used to extract features of the same data type in the initial data set, and the total number of occurrences of each feature and the mean of the total number of occurrences of all features are counted. Features whose total number of occurrences is greater than the mean of the total number of occurrences of all features are marked as the main features of the data type, and features whose total number of occurrences is not greater than the mean of the total number of occurrences of all features are marked as additional features of the data type, and the number of occurrences of the additional features is recorded as the additional feature weight .

[0013] Feature extraction models include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, principal component analysis (PCA), etc.

[0014] Furthermore, the initial data of each data type is screened based on the main features, and the data containing all the main features are classified into the intermediate data set. The number of occurrences of the corresponding additional features of each data in the intermediate data set in different feature extraction models is counted. , combined with additional feature weights , calculate the additional coefficient for each data in the intermediate data set Fj :

[0015] in, j Indicates the sequential number of additional features of the same data. ; The additional coefficients for each data in the intermediate data setFj Greater than The data is divided into the pending data set, Represents the additional coefficient for each data in the intermediate data set Fj The mean 。

[0016] Furthermore, the number of views of each data in the pending data set is obtained through the data source background statistics tool Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz :

[0017] in, Indicates the number of views of all data in the pending dataset Li Mean, Indicates the sharing amount of all data in the pending dataset Fx Mean, Represents the evaluation amount of all data in the pending data set Pj Mean, Indicates real time.

[0018] Furthermore, each data in the pending data set is renumbered and the additional coefficient of each data is extracted. and value coefficient , calculate the typical index of each data in the pending data set :

[0019] in, x Indicates the sequential number of each data in the pending data set. .

[0020] Furthermore, according to the typical index of each data in the undetermined data set The size sorting determines the display priority of each data under each data type, that is, the typical index The larger the data, the higher the priority.

[0021] An online data acquisition and display device, comprising: Data source determination module, input data acquisition requirements, based on the similarity of acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data , determine the data source for obtaining online data; The data screening module obtains the initial data from the online data acquisition data source, uses a variety of different feature extraction models to extract the features of the same data type in the initial data set, divides the main features into additional features, and puts the data containing all the main features into the intermediate data set, and calculates the additional coefficient of each data in the intermediate data set. Fj , the additional coefficient of each data in the intermediate data set Fj Greater than The data is divided into the pending data set; Data display priority determination module, obtains the number of views of each data in the pending data set Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz , based on the additional coefficients for each data and value coefficient Calculate the typical index of each data in the pending data set , determine the display priority of each data under each data type.

[0022] An online data acquisition and display electronic device comprises a processor, a memory, a user interface and a network interface. The memory is used for storing data, the user interface and the network interface are used for communication, and the processor is used for analyzing and issuing instructions.

[0023] (III) Beneficial effects The present invention provides an online data acquisition and display method and device and electronic equipment, which have the following beneficial effects: 1. Input data acquisition requirements, based on the similarity of acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data , determine the data source for acquiring online data this time, predict which data sources may meet current needs more efficiently, and directly reuse or quickly locate the source of previously successfully acquired data, thereby optimizing the resource allocation of the data collection team or system, focusing on high-value data sources, and avoiding repeated searches and verifications of new data sources.

[0024] 2. Obtain initial data from the online data acquisition data source, use a variety of different feature extraction models to extract features of the same data type in the initial data set, divide the main features and additional features, and classify the data containing all the main features into the intermediate data set, and calculate the additional coefficient of each data in the intermediate data set. Fj , the additional coefficient of each data in the intermediate data set Fj Greater than The data can be divided into pending data sets and the data in the data sets can be continuously screened, which can significantly reduce the amount of data to be processed later and realize the refined screening of data. It not only retains the data that may have display value, but also avoids incorporating all the data into subsequent processing, thereby improving processing efficiency.

[0025] 3. Get the number of views of each data in the pending data set Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz , based on the additional coefficients for each data and value coefficient Calculate the typical index of each data in the pending data set , determine the display priority of each data under each data type, and comprehensively evaluate the importance and representativeness of each data under a specific data type, ensuring that the most valuable, representative and fresh data are displayed first with limited display resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of an online data acquisition and display method of the present invention; Figure 2 This is a structural schematic diagram of an online data acquisition and display device of the present invention; Figure 3 The present invention is a schematic structural diagram of an online data acquisition and display electronic device. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1 The present invention provides an online data acquisition and display method, comprising the following steps: Step 1: Input data acquisition requirements based on the similarity of acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data , determine the data source for obtaining the online data.

[0029] The step 1 includes the following contents: Step 101: Input the purpose of data acquisition, such as market analysis, user behavior research, product optimization, etc., and determine the required data type, scope, time span, and subject elements.

[0030] Data type, such as numerical, categorical, and time series; determine the spatial or domain scope of data collection. For example, whether to collect data worldwide or limited to a certain region or country; clarify the start and end time of data collection; Data Theme refers to the core concept or field around which a group of related data revolve.

[0031] Step 102: Use a pre-trained word embedding model (such as Word2Vec, GloVe, BERT, etc.) to convert the words in the acquisition purpose or topic into high-dimensional vectors, aggregate the word vectors in the acquisition purpose or topic (such as average, weighted average, TF-IDF weighting, etc.), obtain the vector representation of the entire acquisition purpose or topic, and then calculate the acquisition purpose Hm of the current input and the acquisition purpose of the acquisition history data The cosine similarity of the vector is recorded as the similarity of the target , and calculate the acquisition theme Hz of the current input and the acquisition theme of the historical data The cosine similarity of the vector is recorded as the topic similarity .in, , .

[0032] Step 103: similarity based on the acquisition target and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data :

[0033] in, i Indicates the time sequence number of all historical data acquisition, .

[0034] Step 104: The comprehensive similarity coefficient Sort from large to small, extract the data sources corresponding to the top 10 historical data, count the number of occurrences of the data sources and sort from large to small, extract the top 3 data sources as the data sources for this online data acquisition.

[0035] When using, combine the contents in steps 101 to 104: Input data acquisition requirements, based on the similarity of acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data , determine the data source for acquiring online data this time, predict which data sources may meet current needs more efficiently, and directly reuse or quickly locate the source of previously successfully acquired data, thereby optimizing the resource allocation of the data collection team or system, focusing on high-value data sources, and avoiding repeated searches and verifications of new data sources.

[0036] Step 2: Get the initial data from the online data source, use a variety of different feature extraction models to extract the features of the same data type in the initial data set, divide the main features and additional features, put the data containing all the main features into the intermediate data set, and calculate the additional coefficient of each data in the intermediate data set. Fj , the additional coefficient of each data in the intermediate data set Fj Greater than The data is divided into the pending data set.

[0037] The step 2 includes the following contents: Step 201: Use a depth-first search algorithm to obtain initial data from the online data acquisition data source based on the required data type, range, time span and subject elements, pre-process the initial data to remove noise, outliers, duplicate values ​​and missing values ​​in the data, and then construct an initial data set.

[0038] Step 202: Use a variety of different feature extraction models to extract features of the same data type in the initial data set, count the total number of occurrences of each feature and the mean of the total number of occurrences of all features, mark the features whose total number of occurrences is greater than the mean of the total number of occurrences of all features as the main features of the data type, and mark the features whose total number of occurrences is not greater than the mean of the total number of occurrences of all features as additional features of the data type, and record the number of occurrences of the additional features as the additional feature weight .

[0039] Feature extraction models include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, principal component analysis (PCA), etc.

[0040] Step 203: Screen the initial data of each data type based on the main features, classify the data containing all the main features into the intermediate data set, and count the number of occurrences of the corresponding additional features of each data in the intermediate data set in different feature extraction models. , combined with additional feature weights , calculate the additional coefficient for each data in the intermediate data set Fj :

[0041] in, jIndicates the sequential number of additional features of the same data. .

[0042] Step 204: Add the additional coefficient of each data in the intermediate data set Fj Greater than The data is divided into the pending data set, Represents the additional coefficient for each data in the intermediate data set Fj The mean 。

[0043] When using, combine the contents in steps 201 to 204: Obtain initial data from the online data acquisition data source, use a variety of different feature extraction models to extract features of the same data type in the initial data set, divide the main features and additional features, divide the data containing all the main features into the intermediate data set, and calculate the additional coefficient of each data in the intermediate data set Fj , the additional coefficient of each data in the intermediate data set Fj Greater than The data can be divided into pending data sets and the data in the data sets can be continuously screened, which can significantly reduce the amount of data to be processed later and realize the refined screening of data. It not only retains the data that may have display value, but also avoids incorporating all the data into subsequent processing, thereby improving processing efficiency.

[0044] Step 3: Get the number of views of each data in the pending data set Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz , based on the additional coefficients for each data and value coefficient Calculate the typical index of each data in the pending data set , determine the display priority of each data under each data type.

[0045] The step three comprises the following steps: Step 301: Obtain the number of views of each data in the pending data set through the data source background statistics tool Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz :

[0046] in, Indicates the number of views of all data in the pending datasetLi Mean, Indicates the sharing amount of all data in the pending dataset Fx Mean, Represents the evaluation amount of all data in the pending data set Pj Mean, Indicates real time.

[0047] Step 302: renumber each data in the pending data set and extract the additional coefficient of each data and value coefficient , calculate the typical index of each data in the pending data set :

[0048] in, x Indicates the sequential number of each data in the pending data set. .

[0049] Step 303: Based on the typical index of each data in the pending data set The size sorting determines the display priority of each data under each data type, that is, the typical index The larger the data, the higher the priority.

[0050] When using, combine the contents in steps 301 to 303: Get the number of views for each data in the pending data set Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz , based on the additional coefficients for each data and value coefficient Calculate the typical index of each data in the pending data set , determine the display priority of each data under each data type, and comprehensively evaluate the importance and representativeness of each data under a specific data type, ensuring that the most valuable, representative and fresh data are displayed first with limited display resources.

[0051] See also Figure 2 The present invention provides an online data acquisition and display device, comprising: Data source determination module, input data acquisition requirements, based on the similarity of acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data , determine the data source for obtaining the online data.

[0052] The data screening module obtains the initial data from the online data acquisition data source, uses a variety of different feature extraction models to extract the features of the same data type in the initial data set, divides the main features into additional features, and puts the data containing all the main features into the intermediate data set, and calculates the additional coefficient of each data in the intermediate data set. Fj , the additional coefficient of each data in the intermediate data set Fj Greater than The data is divided into the pending data set.

[0053] Data display priority determination module, obtains the number of views of each data in the pending data set Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz , based on the additional coefficients for each data and value coefficient Calculate the typical index of each data in the pending data set , determine the display priority of each data under each data type.

[0054] See also Figure 3 The present invention provides an online data acquisition and display electronic device, including a processor, a memory, a user interface and a network interface, the memory is used to store data, the user interface and the network interface are used for communication, and the processor is used to analyze and issue instructions.

[0055] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0056] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0057] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An online data acquisition and display method, Features: The steps include: Input data acquisition requirements, based on the similarity of acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data , determine the data source for obtaining online data; Obtain initial data from the online data acquisition data source, use a variety of different feature extraction models to extract features of the same data type in the initial data set, divide the main features and additional features, divide the data containing all the main features into the intermediate data set, and calculate the additional coefficient of each data in the intermediate data set Fj , the additional coefficient of each data in the intermediate data set Fj Greater than The data is divided into the pending data set; Get the number of views for each data in the pending data set Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz , based on the additional coefficient for each data and value coefficient Calculate the typical index of each data in the pending data set , determine the display priority of each data under each data type.

2. According to the online data acquisition and display method of claim 1, Features: Based on the similarity of acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data : in, i Indicates the time sequence number of all historical data acquisition, .

3. According to the online data acquisition and display method of claim 2, Features: The comprehensive similarity coefficient Sort from large to small, extract the data sources corresponding to the top 10 historical data, count the number of occurrences of the data sources and sort from large to small, extract the top 3 data sources as the data sources for this online data acquisition.

4. According to the online data acquisition and display method of claim 3, Features: Use a variety of different feature extraction models to extract features of the same data type in the initial data set, count the total number of occurrences of each feature and the mean of the total number of occurrences of all features, and mark the features whose total number of occurrences is greater than the mean of the total number of occurrences of all features as the main features of the data type, and the features whose total number of occurrences is not greater than the mean of the total number of occurrences of all features as additional features of the data type, and record the number of occurrences of the additional features as the additional feature weight .

5. According to the online data acquisition and display method of claim 4, Features: The initial data of each data type is screened based on the main features, and the data containing all the main features are classified into the intermediate data set. The number of occurrences of the corresponding additional features of each data in the intermediate data set in different feature extraction models is counted. , combined with additional feature weights , calculate the additional coefficient for each data in the intermediate data set Fj : in, j Indicates the sequential number of additional features of the same data. ; The additional coefficients for each data in the intermediate data set Fj Greater than The data is divided into the pending data set, Represents the additional coefficient for each data in the intermediate data set Fj The mean 。 6. The online data acquisition and display method according to claim 5, Features: Obtain the number of views of each data in the pending data set through the data source background statistics tool Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz : in, Indicates the number of views of all data in the pending dataset Li Mean, Indicates the sharing amount of all data in the pending dataset Fx Mean, Represents the evaluation amount of all data in the pending data set Pj Mean, Indicates real time.

7. The online data acquisition and display method according to claim 6, Features: Renumber each data in the pending data set and extract the additional coefficient of each data and value coefficient , calculate the typical index of each data in the pending data set : in, x Indicates the sequential number of each data in the pending data set. .

8. The online data acquisition and display method according to claim 7, Features: Based on the typical index of each data in the pending data set The size sorting determines the display priority of each data under each data type, that is, the typical index The larger the data, the higher the priority.

9. An online data acquisition and display device, used to implement the method according to any one of claims 1 to 8, Features: include: Data source determination module, input data acquisition requirements, based on the similarity of acquisition purpose and get topic similarity Calculate the comprehensive similarity coefficient between current input and historical data , determine the data source for obtaining online data; The data screening module obtains the initial data from the online data acquisition data source, uses a variety of different feature extraction models to extract the features of the same data type in the initial data set, divides the main features into additional features, and puts the data containing all the main features into the intermediate data set, and calculates the additional coefficient of each data in the intermediate data set. Fj , the additional coefficient of each data in the intermediate data set Fj Greater than The data is divided into the pending data set; Data display priority determination module, obtains the number of views of each data in the pending data set Li , Shared Fx , Evaluation Pj and release time Fb , calculate the value coefficient of each data in the pending data set Jz , based on the additional coefficient for each data and value coefficient Calculate the typical index of each data in the pending data set , determine the display priority of each data under each data type.

10. An online data acquisition and display electronic device, used to implement the method according to any one of claims 1 to 8, Features: It includes a processor, a memory, a user interface and a network interface. The memory is used to store data, the user interface and the network interface are used for communication, and the processor is used to analyze and issue instructions.

Images