Big data processing method and device based on artificial intelligence and electronic equipment

Abstract

The invention provides a big data processing method and device based on artificial intelligence and electronic equipment, and relates to the technical field of data management, and the method comprises the steps: obtaining initial data and a big data center resource allocation task; determining resource matching information between a server and a task in combination with server resource configuration and the big data center resource allocation task; determining a target server for managing the initial data based on the resource matching information; and sharing the initial data through the target server based on a block chain. According to the method, the security, the integrity and the traceability of the data in the sharing process are ensured, the reliability and the credibility of the data are enhanced, and a solid foundation is provided for subsequent application of the data.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a big data processing method, device and electronic equipment based on artificial intelligence. Background Art

[0002] With the rapid development of information technology, big data has become an important force in promoting social progress and corporate development. As all industries actively embrace big data, companies hope to discover new business opportunities and optimize business processes through data analysis and mining. However, the massiveness, diversity and high speed of data have also brought unprecedented challenges to data management and utilization.

[0003] Existing AI-based big data processing methods also have shortcomings in resource allocation and task scheduling, making it difficult to fully utilize server resources and improve data processing efficiency.

[0004] Therefore, a big data processing method, device and electronic device based on artificial intelligence are proposed. Summary of the invention

[0005] This specification provides a big data processing method, device and electronic device based on artificial intelligence, which ensures the security, integrity and traceability of data during the sharing process, enhances the reliability and credibility of the data, and provides a solid foundation for the subsequent application of the data.

[0006] This specification provides a big data processing method based on artificial intelligence, including:

[0007] Obtain initial data and allocate resources to the big data center;

[0008] Determine resource matching information between the server and the task in combination with the server resource configuration and the big data center resource allocation task;

[0009] Determining a target server for managing the initial data based on the resource matching information;

[0010] The initial data is shared through the target server based on the blockchain.

[0011] Optionally, the obtaining of initial data includes:

[0012] Acquire the edge frequency of the initial data;

[0013] Based on the edge frequency, adjusting the parameters of the initial data, and calculating the constraint conditions of the total amount of real-time collection of the initial data;

[0014] Based on the constraint condition of the total amount of real-time collection of the initial data, the shortest transmission path of the initial data is obtained;

[0015] Determining an acquisition objective function of the initial data based on the shortest transmission path of the initial data;

[0016] Based on the acquisition objective function of the initial data, the acquisition of the initial data is completed.

[0017] Optionally, the acquiring the edge frequency of the initial data includes:

[0018]

[0019] Wherein, fb represents the edge frequency of the initial data, x u represents the confidence of the initial data, f(x) represents the smoothing function of the initial data, μ b represents the edge variance, g(ω) represents the edge frequency function, f′ represents the edge frequency variance of the initial data, c g represents the interference error of the initial data acquisition, c l Represents a filter.

[0020] Optionally, adjusting the parameters of the initial data based on the edge frequency and calculating the constraint conditions of the total amount of real-time collection of the initial data includes:

[0021] Adjusting the parameters of the initial data based on the edge frequency includes:

[0022]

[0023] Among them, ε w represents the parameters of the initial data, e 0 represents the redundant nature of the initial data, ω e represents the weight of the initial data, ρ r Represents the density index of redundant features, r e represents the initial data without redundant features, λ s represents the threshold set in advance, λ r represents a recognition threshold of the initial data;

[0024] Based on the parameters of the initial data, the constraint conditions of the total amount of real-time collection of the initial data are calculated, including:

[0025]

[0026] Among them, y r represents the constraint condition of the total amount of real-time collection of the initial data, d j represents the distance between the initial data and the acquisition node, μ l represents the variance between two adjacent acquisition nodes, vt represents the transmission speed of the initial data, v s represents the acquisition speed of the initial data, d p Indicates the transmission distance between the initial data.

[0027] Optionally, obtaining the shortest transmission path of the initial data based on the constraint condition of the total amount of real-time collection of the initial data includes:

[0028]

[0029] Among them, W r represents the shortest transmission path of the initial data, C s represents the acquisition window of the initial data, B s represents the collection list of the initial data, T represents the data collection period, F d represents the amplitude of the initial data, represents the variation parameter of the initial data, A s Indicates the amplitude change value of the initial data.

[0030] Optionally, determining the acquisition objective function of the initial data based on the shortest transmission path of the initial data includes:

[0031]

[0032] Wherein, φ(x) represents the acquisition objective function of the initial data, f c represents the frequency of the initial data, represents the mean value of the acquisition window, t u represents the time interval for collecting the initial data, Q represents the relationship between the collection node and the number of hops of the initial data, and Z f represents f c The number of hops of the initial data described below.

[0033] Optionally, combining the server resource configuration and the big data center resource allocation task to determine resource matching information between the server and the task includes:

[0034]

[0035] Among them, F(s i , t j ) indicates resource matching information, d jk Represents the resource allocation task t of the big data center j The demand for resource type k, r ik Indicates server i For the availability of resource type k, β represents the adjustment factor.

[0036] This specification provides a big data processing device based on artificial intelligence, including

[0037] The acquisition module is used to obtain initial data and allocate resources to the big data center;

[0038] A determination module, used to determine resource matching information between the server and the task in combination with the server resource configuration and the big data center resource allocation task;

[0039] A management module, configured to determine a target server for managing the initial data based on the resource matching information;

[0040] A sharing module is used to share the initial data through the target server based on the blockchain.

[0041] Optionally, the acquisition module includes:

[0042] Acquire the edge frequency of the initial data;

[0043] Based on the edge frequency, adjusting the parameters of the initial data, and calculating the constraint conditions of the total amount of real-time collection of the initial data;

[0044] Based on the constraint condition of the total amount of real-time collection of the initial data, the shortest transmission path of the initial data is obtained;

[0045] Determining an acquisition objective function of the initial data based on the shortest transmission path of the initial data;

[0046] Based on the acquisition objective function of the initial data, the acquisition of the initial data is completed.

[0047] Optionally, the acquiring the edge frequency of the initial data includes:

[0048]

[0049] Among them, f b represents the edge frequency of the initial data, x u represents the confidence of the initial data, f(x) represents the smoothing function of the initial data, μ b represents the edge variance, g(ω) represents the edge frequency function, f′ represents the edge frequency variance of the initial data, c g represents the interference error of the initial data acquisition, c l Represents a filter.

[0050] Optionally, adjusting the parameters of the initial data based on the edge frequency and calculating the constraint conditions of the total amount of real-time collection of the initial data includes:

[0051] Adjusting the parameters of the initial data based on the edge frequency includes:

[0052]

[0053] Among them, ε w represents the parameters of the initial data, e 0 represents the redundant nature of the initial data, ω e represents the weight of the initial data, ρ r Represents the density index of redundant features, r e represents the initial data without redundant features, λ s represents the threshold set in advance, λ r represents a recognition threshold of the initial data;

[0054] Based on the parameters of the initial data, the constraint conditions of the total amount of real-time collection of the initial data are calculated, including:

[0055]

[0056] Among them, y r represents the constraint condition of the total amount of real-time collection of the initial data, d j represents the distance between the initial data and the acquisition node, μ l represents the variance between two adjacent acquisition nodes, v t represents the transmission speed of the initial data, v s represents the acquisition speed of the initial data, d p Indicates the transmission distance between the initial data.

[0057] Optionally, obtaining the shortest transmission path of the initial data based on the constraint condition of the total amount of real-time collection of the initial data includes:

[0058]

[0059] Among them, W r represents the shortest transmission path of the initial data, C s represents the acquisition window of the initial data, B s represents the collection list of the initial data, T represents the data collection period, F d represents the amplitude of the initial data, represents the variation parameter of the initial data, A s Indicates the amplitude change value of the initial data.

[0060] Optionally, determining the acquisition objective function of the initial data based on the shortest transmission path of the initial data includes:

[0061]

[0062] Wherein, φ(x) represents the acquisition objective function of the initial data, fc represents the frequency of the initial data, represents the mean value of the acquisition window, t u represents the time interval for collecting the initial data, Q represents the relationship between the collection node and the number of hops of the initial data, and Z f represents f c The number of hops of the initial data described below.

[0063] Optionally, the determining module includes:

[0064]

[0065] Among them, F(s i , t j ) indicates resource matching information, d jk Represents the resource allocation task t of the big data center j The demand for resource type k, r ik Indicates server i For the availability of resource type k, β represents the adjustment factor.

[0066] This specification also provides an electronic device, wherein the electronic device includes:

[0067] processor; and,

[0068] A memory storing computer executable instructions, which when executed cause the processor to perform any of the above methods.

[0069] The present specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, any of the above methods is implemented.

[0070] In this specification, the accuracy and completeness of the data are ensured by obtaining the edge frequency of the initial data and adjusting the data parameters accordingly. On this basis, the constraints of the total amount of real-time collection are calculated, providing clear guidance for the collection and transmission of data. By determining the shortest transmission path and the collection objective function, the data collection and transmission process is further optimized, reducing costs and improving efficiency. In addition, the method combines server resource configuration and big data center resource allocation tasks, and achieves the best match between servers and tasks through precise resource matching information. This not only improves resource utilization, but also ensures the smooth completion of tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0072] Figure 1 A schematic diagram of the principle of a big data processing method based on artificial intelligence provided in an embodiment of this specification;

[0073] Figure 2 A schematic diagram of the principle of a big data processing device based on artificial intelligence provided in an embodiment of this specification;

[0074] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification;

[0075] Figure 4 A schematic diagram of a computer-readable medium provided for an embodiment of this specification. DETAILED DESCRIPTION

[0076] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0077] The following is combined with Figure 1-4The exemplary embodiments of the present invention are described more fully. However, the exemplary embodiments can be implemented in various forms, and it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, providing these exemplary embodiments can make the present invention more comprehensive and complete, and it is more convenient to fully convey the inventive concept to those skilled in the art. The same reference numerals in the figures represent the same or similar elements, components or parts, and thus their repeated description will be omitted.

[0078] Under the premise of being consistent with the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.

[0079] In the description of specific embodiments, the features, structures, characteristics or other details described in the present invention are intended to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can practice the technical solutions of the present invention without one or more of the specific features, structures, characteristics or other details.

[0080] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0081] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0082] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0083] Figure 1 A schematic diagram of a big data processing method based on artificial intelligence provided in an embodiment of this specification, the method may include:

[0084] S110: Acquisition of initial data and big data center resource allocation tasks;

[0085] Optionally, the S110 includes:

[0086] Acquire the edge frequency of the initial data;

[0087] Based on the edge frequency, adjusting the parameters of the initial data, and calculating the constraint conditions of the total amount of real-time collection of the initial data;

[0088] Based on the constraint condition of the total amount of real-time collection of the initial data, the shortest transmission path of the initial data is obtained;

[0089] Determining an acquisition objective function of the initial data based on the shortest transmission path of the initial data;

[0090] Based on the acquisition objective function of the initial data, the acquisition of the initial data is completed.

[0091] Optionally, the acquiring the edge frequency of the initial data includes:

[0092]

[0093] Wherein, fb represents the edge frequency of the initial data, x u represents the confidence of the initial data, f(x) represents the smoothing function of the initial data, μ b represents the edge variance, g(ω) represents the edge frequency function, f′ represents the edge frequency variance of the initial data, c g represents the interference error of the initial data acquisition, c l Represents a filter.

[0094] Optionally, adjusting the parameters of the initial data based on the edge frequency and calculating the constraint conditions of the total amount of real-time collection of the initial data includes:

[0095] Adjusting the parameters of the initial data based on the edge frequency includes:

[0096]

[0097] Among them, ε w represents the parameters of the initial data, e 0 represents the redundant nature of the initial data, ω e represents the weight of the initial data, ρ r Represents the density index of redundant features, r e represents the initial data without redundant features, λ s represents the threshold set in advance, λ r represents a recognition threshold of the initial data;

[0098] Based on the parameters of the initial data, the constraint conditions of the total amount of real-time collection of the initial data are calculated, including:

[0099]

[0100] Among them, y rrepresents the constraint condition of the total amount of real-time collection of the initial data, d j represents the distance between the initial data and the acquisition node, μ l represents the variance between two adjacent acquisition nodes, v t represents the transmission speed of the initial data, v s represents the acquisition speed of the initial data, d p Indicates the transmission distance between the initial data.

[0101] Optionally, obtaining the shortest transmission path of the initial data based on the constraint condition of the total amount of real-time collection of the initial data includes:

[0102]

[0103] Among them, W r represents the shortest transmission path of the initial data, C s represents the acquisition window of the initial data, B s represents the collection list of the initial data, T represents the data collection period, F d represents the amplitude of the initial data, represents the variation parameter of the initial data, A s Indicates the amplitude change value of the initial data.

[0104] Optionally, determining the acquisition objective function of the initial data based on the shortest transmission path of the initial data includes:

[0105]

[0106] Wherein, φ(x) represents the acquisition objective function of the initial data, fc represents the frequency of the initial data, represents the mean value of the acquisition window, t u represents the time interval for collecting the initial data, Q represents the relationship between the collection node and the number of hops of the initial data, and Z f represents f c The number of hops of the initial data described below.

[0107] Optionally, completing the collection of the initial data based on the collection objective function of the initial data includes:

[0108]

[0109] Wherein, S represents the collection amount of the initial data, and M represents the total amount of the collected data of the initial data in a single cycle.

[0110] S120: Determine resource matching information between the server and the task in combination with the server resource configuration and the big data center resource allocation task;

[0111] Optionally, the S120 includes:

[0112]

[0113] Among them, F(s i , t j ) indicates resource matching information, d jk Represents the resource allocation task t of the big data center j The demand for resource type k, r ik Indicates server i For the availability of resource type k, β represents the adjustment factor.

[0114] S130: Determine a target server that manages the initial data based on the resource matching information;

[0115] In the specific implementation of this specification, a resource list of all available servers is obtained, which includes the hardware configuration (such as CPU, memory, hard disk, etc.), software configuration (such as operating system, database, middleware, etc.) and network configuration (such as bandwidth, delay, etc.) of the server. According to the characteristics of the initial data (such as data type, data volume, processing requirements, etc.) and business requirements (such as real-time, reliability, security, etc.), the resource list is screened and matched to find servers that meet the requirements. After screening out the servers that meet the requirements, the performance of these servers needs to be evaluated. This can be done by simulating the relevant tasks of running the initial data and observing the server's response time, throughput, resource utilization and other indicators. In addition to performance evaluation, cost-effectiveness also needs to be considered. This includes the purchase cost, operation and maintenance cost, energy consumption cost, etc. of the server. In the case of comparable performance, servers with higher cost-effectiveness are preferred. If there are multiple servers that meet the requirements, the problem of load balancing also needs to be considered. Through a reasonable load balancing strategy, it is possible to ensure that server resources are fully utilized and avoid idle or overloaded resources. Finally, the security and reliability of the server also need to be considered. This includes the physical security, network security, data backup and recovery strategies, etc. of the server. Ensure that the target server can provide stable and secure data management services.

[0116] S140: Sharing the initial data through the target server based on blockchain.

[0117] In the specific implementation of this specification, a blockchain network is constructed according to data sharing requirements, including determining the type of blockchain (such as public chain, private chain or alliance chain), consensus mechanism, smart contract, etc. The initialization generator is connected to generate the initial state and parameters of the blockchain network. The target server is connected to the blockchain network to ensure that the server can communicate and exchange data with other nodes in the blockchain network. The necessary data processing and data sharing services are deployed on the target server, such as data indexing service, data access control service, etc.

[0118] The data owner uploads the data index and encrypted data to the target server. The target server, as a cloud storage center, is responsible for storing and managing this data. After the data is successfully uploaded, the target server generates a data index list for subsequent data access and sharing. The data owner formulates a data sharing strategy based on business needs, including data access rights, access control rules, etc. The sharing strategy is deployed on the blockchain network in the form of a smart contract to ensure that the execution and verification process of the strategy is automated and transparent.

[0119] The data requester sends a data access request to the target server and uploads the data index to be accessed. The target server searches for the corresponding encrypted data in the local storage based on the data index. The target server forwards the data access request to the blockchain network, and the smart contract in the blockchain network verifies the authority. After the verification, the blockchain network generates and issues an access key to the data requester. The access key exists in encrypted form to ensure security during transmission. The data requester uses the access key to decrypt the encrypted data and obtain the original data. The data requester can access, process, and analyze the data within the scope of authorization.

[0120] Throughout the data sharing process, data is always in encrypted form to ensure data privacy and security. Use the tamper-proof features of blockchain to ensure the integrity and authenticity of data during the sharing process. Implement automated access control through smart contracts to ensure that only authorized users can access data. Record log information of data access, including access time, access user, access content, etc., for subsequent auditing and tracing. Comply with relevant laws, regulations and industry standards to ensure the legality and compliance of the data sharing process. Establish a data sharing supervision mechanism to monitor and manage data sharing activities in real time.

[0121] In this specification, the accuracy and completeness of the data are ensured by obtaining the edge frequency of the initial data and adjusting the data parameters accordingly. On this basis, the constraints of the total amount of real-time collection are calculated, providing clear guidance for the collection and transmission of data. By determining the shortest transmission path and the collection objective function, the data collection and transmission process is further optimized, reducing costs and improving efficiency. In addition, the method combines server resource configuration and big data center resource allocation tasks, and achieves the best match between servers and tasks through precise resource matching information. This not only improves resource utilization, but also ensures the smooth completion of tasks.

[0122] Figure 2 A schematic diagram of a big data processing device based on artificial intelligence provided in an embodiment of this specification may include:

[0123] An acquisition module 10 is used to acquire initial data and big data center resource allocation tasks;

[0124] A determination module 20, for determining resource matching information between the server and the task in combination with the server resource configuration and the big data center resource allocation task;

[0125] A management module 30, configured to determine a target server for managing the initial data based on the resource matching information;

[0126] The sharing module 40 is used to share the initial data through the target server based on the blockchain.

[0127] Optionally, the acquisition module 10 includes:

[0128] Acquire the edge frequency of the initial data;

[0129] Based on the edge frequency, adjusting the parameters of the initial data, and calculating the constraint conditions of the total amount of real-time collection of the initial data;

[0130] Based on the constraint condition of the total amount of real-time collection of the initial data, the shortest transmission path of the initial data is obtained;

[0131] Determining an acquisition objective function of the initial data based on the shortest transmission path of the initial data;

[0132] Based on the acquisition objective function of the initial data, the acquisition of the initial data is completed.

[0133] Optionally, the acquiring the edge frequency of the initial data includes:

[0134]

[0135] Wherein, fb represents the edge frequency of the initial data, x u represents the confidence of the initial data, f (x) represents the smoothing function of the initial data, μ b represents the edge variance, g(ω) represents the edge frequency function, f′ represents the edge frequency variance of the initial data, c g represents the interference error of the initial data acquisition, c l Represents a filter.

[0136] Optionally, adjusting the parameters of the initial data based on the edge frequency and calculating the constraint conditions of the total amount of real-time collection of the initial data includes:

[0137] Adjusting the parameters of the initial data based on the edge frequency includes:

[0138]

[0139] Among them, ε w represents the parameters of the initial data, e 0 represents the redundant nature of the initial data, ω e represents the weight of the initial data, ρ r Represents the density index of redundant features, r e represents the initial data without redundant features, λ s represents the threshold set in advance, λ r represents a recognition threshold of the initial data;

[0140] Based on the parameters of the initial data, the constraint conditions of the total amount of real-time collection of the initial data are calculated, including:

[0141]

[0142] Among them, y r represents the constraint condition of the total amount of real-time collection of the initial data, d j represents the distance between the initial data and the acquisition node, μ l represents the variance between two adjacent acquisition nodes, v t represents the transmission speed of the initial data, v s represents the acquisition speed of the initial data, d p Indicates the transmission distance between the initial data.

[0143] Optionally, obtaining the shortest transmission path of the initial data based on the constraint condition of the total amount of real-time collection of the initial data includes:

[0144]

[0145] Among them, W r represents the shortest transmission path of the initial data, C s represents the acquisition window of the initial data, B s represents the collection list of the initial data, T represents the data collection period, F d represents the amplitude of the initial data, represents the variation parameter of the initial data, A s Indicates the amplitude change value of the initial data.

[0146] Optionally, determining the acquisition objective function of the initial data based on the shortest transmission path of the initial data includes:

[0147]

[0148] Wherein, φ(x) represents the acquisition objective function of the initial data, f c represents the frequency of the initial data, represents the mean value of the acquisition window, t u represents the time interval for collecting the initial data, Q represents the relationship between the collection node and the number of hops of the initial data, and Z f represents f c The number of hops of the initial data described below.

[0149] Optionally, the determining module 30 includes:

[0150]

[0151] Among them, F(s i , t j ) indicates resource matching information, d jk Represents the resource allocation task t of the big data center j The demand for resource type k, r ik Indicates server i For the availability of resource type k, β represents the adjustment factor.

[0152] The functions of the device in the embodiment of the present invention have been described in the above method embodiment, so for details not covered in the description of this embodiment, please refer to the relevant description in the above embodiment, and no further description will be given here.

[0153] Based on the same inventive concept, an embodiment of this specification also provides an electronic device.

[0154] The following describes an electronic device embodiment of the present invention, which can be regarded as a specific physical implementation of the method and device embodiments of the present invention. The details described in the electronic device embodiment of the present invention should be regarded as a supplement to the above method or device embodiments; details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above method or device embodiments.

[0155] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Figure 3 The electronic device 300 according to the embodiment of the present invention is described. Figure 3 The electronic device 300 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0156] like Figure 3 As shown, the electronic device 300 is in the form of a general computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including the storage unit 320 and the processing unit 310), a display unit 340, etc.

[0157] The storage unit stores program codes, which can be executed by the processing unit 310, so that the processing unit 310 performs the steps according to various exemplary embodiments of the present invention described in the above processing method section of this specification. For example, the processing unit 310 can perform the following steps: Figure 1 Steps shown.

[0158] The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit 3202 , and may further include a read-only memory unit (ROM) 3203 .

[0159] The storage unit 320 may also include a program / utility 3204 having a set (at least one) of program modules 3205, such program modules 3205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0160] Bus 330 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0161] The electronic device 300 may also communicate with one or more external devices 400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable viewers to interact with the electronic device 300, and / or any device that enables the electronic device 300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 350. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 360. The network adapter 360 may communicate with other modules of the electronic device 300 through the bus 330. It should be understood that although Figure 3 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0162] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present invention can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation method of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, including a number of instructions to enable a computing device (which can be a personal computer, server, or network device, etc.) to execute the above method according to the present invention. When the computer program is executed by a data processing device, the computer-readable medium can implement the above method of the present invention, that is: Figure 1 The method shown.

[0163] Figure 4 A schematic diagram of a computer-readable medium provided for an embodiment of this specification.

[0164] accomplish Figure 1The computer program of the method shown can be stored on one or more computer readable media. The computer readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0165] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0166] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the viewer computing device, partially on the viewer device, as a stand-alone software package, partially on the viewer computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the viewer computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0167] In summary, the present invention can be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that general data processing devices such as microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0168] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0169] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0170] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A big data processing method based on artificial intelligence, characterized in that: include: Obtain initial data and allocate resources to the big data center; Determine resource matching information between the server and the task in combination with the server resource configuration and the big data center resource allocation task; Determining a target server for managing the initial data based on the resource matching information; The initial data is shared through the target server based on the blockchain.

2. The big data processing method based on artificial intelligence according to claim 1, characterized in that: The obtaining of initial data comprises: Acquire the edge frequency of the initial data; Based on the edge frequency, adjusting the parameters of the initial data, and calculating the constraint conditions of the total amount of real-time collection of the initial data; Based on the constraint condition of the total amount of real-time collection of the initial data, the shortest transmission path of the initial data is obtained; Determining an acquisition objective function of the initial data based on the shortest transmission path of the initial data; Based on the acquisition objective function of the initial data, the acquisition of the initial data is completed.

3. The big data processing method based on artificial intelligence according to claim 2, characterized in that: The obtaining the edge frequency of the initial data comprises: Among them, f b represents the edge frequency of the initial data, x u represents the confidence of the initial data, f(x) represents the smoothing function of the initial data, μ b represents the edge variance, g(ω) represents the edge frequency function, and f ′ represents the edge frequency variance of the initial data, c g represents the interference error of the initial data acquisition, c l Represents a filter.

4. The big data processing method based on artificial intelligence according to claim 3, characterized in that: The adjusting the parameters of the initial data based on the edge frequency and calculating the constraint conditions of the total amount of real-time collection of the initial data include: Adjusting the parameters of the initial data based on the edge frequency includes: Among them, ε w represents the parameters of the initial data, e0 represents the redundant characteristics of the initial data, ω e represents the weight of the initial data, ρ r Represents the density index of redundant features, r e represents the initial data without redundant features, λ s represents the threshold set in advance, λ r represents a recognition threshold of the initial data; Based on the parameters of the initial data, the constraint conditions of the total amount of real-time collection of the initial data are calculated, including: Among them, y r represents the constraint condition of the total amount of real-time collection of the initial data, d j represents the distance between the initial data and the acquisition node, μ l represents the variance between two adjacent acquisition nodes, v t represents the transmission speed of the initial data, v s represents the acquisition speed of the initial data, d p Indicates the transmission distance between the initial data.

5. The big data processing method based on artificial intelligence according to claim 4, characterized in that: The obtaining of the shortest transmission path of the initial data based on the constraint condition of the total amount of real-time collection of the initial data comprises: Among them, W r represents the shortest transmission path of the initial data, C s represents the acquisition window of the initial data, B s represents the collection list of the initial data, T represents the data collection period, F d represents the amplitude of the initial data, represents the variation parameter of the initial data, A s Indicates the amplitude change value of the initial data.

6. The big data processing method based on artificial intelligence according to claim 5, characterized in that: The determining of the acquisition objective function of the initial data based on the shortest transmission path of the initial data includes: Wherein, φ(x) represents the acquisition objective function of the initial data, f c represents the frequency of the initial data, represents the mean value of the acquisition window, t u represents the time interval for collecting the initial data, Q represents the relationship between the collection node and the number of hops of the initial data, and Z f represents f c The number of hops of the initial data described below.

7. The big data processing method based on artificial intelligence according to claim 6, characterized in that: The combining of the server resource configuration and the big data center resource allocation task to determine the resource matching information between the server and the task includes: Among them, F(s i ,t j ) indicates resource matching information, d jk Represents the resource allocation task t of the big data center j The demand for resource type k, r ik Indicates server i For the availability of resource type k, β represents the adjustment factor.

8. A big data processing method based on artificial intelligence, characterized in that: include: The acquisition module is used to obtain initial data and allocate resources to the big data center; A determination module, used to determine resource matching information between the server and the task in combination with the server resource configuration and the big data center resource allocation task; A management module, configured to determine a target server for managing the initial data based on the resource matching information; A sharing module is used to share the initial data through the target server based on the blockchain.

9. An electronic device, wherein: The electronic device includes: processor; and, A memory storing computer executable instructions which, when executed, cause the processor to perform a method according to any one of claims 1-7.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method of any one of claims 1 to 7 is implemented.

Images