Data association method and device

Abstract

The invention discloses a data association method and device, and relates to the technical field of big data. A specific embodiment of the method comprises the following steps: in response to obtaining to-be-associated data from a message queue, judging whether the to-be-associated data obtained in a current period contains all the to-be-associated data or not; if not, determining missing to-be-associated data, selecting target data having an association relationship with the missing to-be-associated data from the to-be-associated data acquired in the current period, and storing the target data in a preset cache; and in response to missing to-be-associated data received in a later period, reading the target data from the preset cache, and associating the missing to-be-associated data with the target data. According to the embodiment, the problem that association cannot be carried out due to successive arrival of the to-be-associated data can be solved on the premise of not increasing the operand and not influencing the calculation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data association method and device. Background Art

[0002] In the real-time processing of big data, it is often necessary to associate data of different business types in the message queue. For example, in the logistics scenario, it is necessary to associate an order data and multiple waybill data with an associated relationship in KAFKA (a message queue). Through the association, three associated result data can be formed. Each associated result data contains the order data and a waybill data. The above associated result data can be used for subsequent delivery fulfillment links.

[0003] At present, due to network delays and other reasons, the data to be associated, such as the order data and waybill data mentioned above, may not arrive in the same time period, resulting in the inability to perform data association. In the prior art, a sliding window is generally used to select data multiple times to cover the data to be associated within a certain period of time, in order to solve the problem of the data to be associated arriving in sequence. However, the execution parameters of the above method need to be set based on experience, and there is still the problem of not being able to obtain the full amount of data to be associated. In addition, the above method will select duplicate data, which has the risk of affecting the accuracy of the calculation results. In some cases, it is necessary to perform data deduplication steps to increase the amount of calculation. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a data association method and device, which can solve the problem of being unable to associate due to the sequential arrival of data to be associated without increasing the amount of calculation and affecting the calculation accuracy.

[0005] To achieve the above objective, according to one aspect of the present invention, a data association method is provided.

[0006] The data association method of the embodiment of the present invention is used to associate multiple pieces of to-be-associated data belonging to multiple preset types to form at least one piece of associated result data; any associated result data is formed by multiple pieces of to-be-associated data with pre-established association relationships; the method is executed by a consumer of a preset message queue, including: in response to obtaining the to-be-associated data from the message queue, judging whether the to-be-associated data obtained in the current cycle contains all the to-be-associated data; if not, determining the missing to-be-associated data, selecting target data having an association relationship with the missing to-be-associated data from the to-be-associated data obtained in the current cycle, and storing the target data in a preset cache; in response to receiving the missing to-be-associated data in a later cycle, reading the target data from the preset cache, and associating the missing to-be-associated data with the target data.

[0007] Optionally, the method further comprises: in response to the expiration of the current cycle, associating the to-be-associated data having an association relationship among the to-be-associated data acquired in the current cycle.

[0008] Optionally, in response to receiving the missing data to be associated in a later period, reading the target data from the preset cache, and associating the missing data to be associated with the target data, includes: in response to receiving all of the missing data to be associated in the later period, reading the target data from the preset cache, and associating all of the missing data to be associated with the target data; the method further includes: after the association is completed, deleting all target data in the preset cache.

[0009] Optionally, the method further includes: in response to receiving part of the missing data to be associated in the later period, determining the currently missing data to be associated; reading specific data that has an association relationship with the part of the missing data to be associated from the preset cache, and associating the specific data with the part of the missing data to be associated; retaining the target data in the target data that has an association relationship with the currently missing data to be associated in the preset cache, and deleting the target data other than the target data from the preset cache.

[0010] Optionally, the message queue adopts a subscription and publishing mode, and the data to be associated of the same preset type is in an independent topic of the message queue; and the method further includes: after obtaining the associated result data, sending the associated result data to an independent topic of the message queue to execute the transmission of the associated result data.

[0011] Optionally, each preset type includes at least one piece of data to be associated, each piece of associated result data corresponds to the plurality of preset types of data to be associated, and the consumer is a stream computing engine.

[0012] To achieve the above objective, according to another aspect of the present invention, a data association device is provided.

[0013] The data association device of the embodiment of the present invention is used to associate multiple pieces of to-be-associated data belonging to multiple preset types to form at least one piece of association result data; any association result data is formed by multiple pieces of to-be-associated data with pre-established association relationships; the device is set at a consumer of a preset message queue, including: a judgment unit, used to respond to obtaining the to-be-associated data from the message queue, and judge whether the to-be-associated data obtained in the current cycle contains all the to-be-associated data; a cache unit, used to determine the missing to-be-associated data when it is judged that it is not included, select target data that has an association relationship with the missing to-be-associated data from the to-be-associated data obtained in the current cycle, and store the target data in the preset cache; an association unit, used to respond to receiving the missing to-be-associated data in a later cycle, read the target data from the preset cache, and associate the missing to-be-associated data with the target data.

[0014] Optionally, the associating unit is further configured to: in response to the expiration of the current period, associate the to-be-associated data having an associative relationship among the to-be-associated data acquired in the current period.

[0015] To achieve the above objective, according to another aspect of the present invention, an electronic device is provided.

[0016] An electronic device of the present invention comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data association method provided by the present invention.

[0017] To achieve the above objective, according to yet another aspect of the present invention, a computer-readable storage medium is provided.

[0018] A computer-readable storage medium of the present invention stores a computer program, which implements the data association method provided by the present invention when executed by a processor.

[0019] According to the technical solution of the present invention, the embodiments of the above invention have the following advantages or beneficial effects:

[0020] After obtaining a piece of data to be associated from the message queue, determine whether the data to be associated obtained in the current cycle contains all the data to be associated: if not, determine the missing data to be associated, and select the target data that has an association relationship with the missing data to be associated from the data to be associated obtained in the current cycle, and store the target data in the preset cache. Thereafter, when the missing data to be associated is received, the target data is read from the preset cache, and the missing data to be associated is associated with the target data, thereby achieving the correct association of multiple preset types of data to be associated, solving the problem of being unable to be associated due to the arrival of the data to be associated one after another. The above solution does not require the use of a sliding window, does not increase the amount of calculation, and does not affect the calculation accuracy.

[0021] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0023] Figure 1 Schematic diagram of main steps of the data association method in an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of the system architecture of the first embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of the execution steps of the first embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of components of a data association device in an embodiment of the present invention;

[0027] Figure 5 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0028] Figure 6 It is a schematic diagram of the structure of an electronic device used to implement the data association method in the embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] The data association method of the present invention is used to associate multiple pieces of data to be associated, thereby forming at least one piece of associated result data. The above multiple pieces of data to be associated refer to a group of data that have an associated relationship with each other. These pieces of data to be associated belong to multiple preset types respectively. Each associated result data finally formed is formed by multiple pieces of data to be associated that have established an associated relationship in advance, and covers all preset types. The above preset types can be data types at the business level, and each preset type contains at least one piece of data to be associated. For example, in e-commerce and logistics scenarios, there is a correlation between the orders and waybills of the same user. If an order corresponds to three waybills, then the corresponding order data a and three waybill data b, c, and d form the above data to be associated, and the order data and waybill data are preset types. Through data association, the order data a can be associated with the waybill data b to form the associated result data e, the order data a can be associated with the waybill data c to form the associated result data f, and the order data a can be associated with the waybill data d to form the associated result data g. Then it is said that there is an associated relationship between a and b, an associated relationship between a and c, and an associated relationship between a and d. The above association relationship is predetermined, and can be represented as a correspondence between an order ID and three waybill IDs, for example. It can be seen that in a group of data to be associated, a preset type can include at least one piece of data to be associated.

[0031] The present invention can be applied to relevant scenarios of message queues and executed by a consumer of the message queue. The consumer can be a stream computing engine in the big data field, such as Spark-Streaming.

[0032] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0033] Figure 1 Schematic diagram of main steps of the data association method according to an embodiment of the present invention.

[0034] like Figure 1 As shown, the data association method of the embodiment of the present invention can be specifically performed according to the following steps:

[0035] Step S101: in response to obtaining the data to be associated from the message queue, determining whether the data to be associated obtained in the current cycle includes all the data to be associated.

[0036] In this step, the consumer of the message queue consumes messages from the message queue, thereby obtaining the data in the message. The above message queue can be a message queue in point-to-point mode or a message queue in subscription and publication mode. When the message queue adopts the subscription and publication mode, the to-be-associated data of the same preset type can be stored in an independent topic of the message queue, and the to-be-associated data of different preset types are in different topics, which is conducive to data management in the message queue and data acquisition by consumers, and avoids data confusion.

[0037] When a consumer receives the first piece of data in a group of data to be associated, it can determine whether the data to be associated obtained in the current cycle contains all the data to be associated after the current cycle expires. If not, execute step S102. If it is already included, it means that a full set of data to be associated has been obtained in the current cycle, and then the association can be performed directly on these data to be associated to obtain the associated result data. After obtaining the associated result data, the consumer can send the associated result data to an independent topic of the message queue (which may be the same or different from the previous message queue) (provided that the message queue adopts the subscription and publishing mode) to execute the transmission of the associated result data in subsequent links. The above independent topics are different from the message queue topics where the data to be associated is located. The above period refers to the time range of the fixed time span set internally by the consumer. The time span of the period is generally set to be equal, and the consumer can directly calculate and process the data obtained in the same period.

[0038] Step S102: Determine the missing data to be associated, select target data that is associated with the missing data to be associated from the data to be associated acquired in the current cycle, and store the target data in a preset cache.

[0039] In this step, if the consumer does not receive the entire set of data to be associated in the current cycle, the missing data to be associated is first determined, that is, the data in a set of data to be associated except for the data already acquired in the current cycle, and then the target data that has an association relationship with the missing data to be associated is selected from the data to be associated acquired in the current cycle, and the target data is stored in the preset cache. It can be understood that in order to normally associate some of the delayed data to be associated, it is necessary to cache the existing data that has an association relationship with these data to be associated, and then read the target data from the cache to perform the association after these missing data to be associated arrive. The above preset cache can be a storage unit in a memory or a hard disk. Exemplarily, the preset cache is Redis for facilitating data query.

[0040] In actual applications, after the current cycle ends, the consumer can associate the data to be associated with the associated data obtained in the current cycle, and send the obtained associated result data to an independent topic of the message queue to execute the transmission of the associated result data in the subsequent links. The above association step and the target data caching step can be performed simultaneously or in any order.

[0041] Step S103: in response to receiving the missing data to be associated in the later period, reading the target data from the preset cache, and associating the missing data to be associated with the target data.

[0042] In this step, if the consumer receives all the missing data to be associated in a later cycle after the current cycle, the target data can be read from the preset cache, and all the missing data to be associated are associated with the target data to obtain the associated result data. After obtaining the associated result data, the consumer can send the associated result data to an independent topic of the message queue to execute the transmission of the associated result data in the subsequent links, thereby completing the association of a group of data to be associated. After the association is completed, the consumer can delete all the target data in the preset cache. Since the consumer can determine the association relationship between a group of data to be associated in advance, such as the identifiers in each data to be associated, the target data can be found and read from the preset cache by the identifier of the target data.

[0043] In another case, if the consumer receives only part of the missing data to be associated in the above later period, on the one hand, the consumer can read the specific data associated with the part of the missing data to be associated from the preset cache, associate the specific data with the part of the missing data to be associated, and send the obtained associated result data to an independent topic of the message queue to execute the transmission of the associated result data in the subsequent links. On the other hand, the consumer determines the currently missing data to be associated, that is, the data that has not been obtained so far in a group of data to be associated, and then retains the target data in the target data that has an associated relationship with the currently missing data to be associated in the preset cache, and deletes the target data other than the target data from the preset cache. It can be understood that the above target data is used to perform association when the remaining data to be associated is received later, and the target data other than the target data is not needed for the data association of Houxi, so these data are deleted from the preset cache.

[0044] Finally, after receiving all the remaining data in a certain cycle, the consumer can read the target data from the preset cache to perform data association, and send the obtained association result data to an independent topic of the message queue to perform the transmission of the association result data in the subsequent links, thereby completing the association processing of a set of data to be associated. Of course, if all the remaining data cannot be obtained in a certain cycle, some of the data that needs to be used later in the preset cache can be kept in the cache in a similar way, the other data in the cache can be deleted, and the arrival of the last remaining data can be waited for. In actual scenarios, a certain waiting time threshold can be set, and an alarm can be executed when the waiting time exceeds the threshold.

[0045] In the technical solution of the embodiment of the present invention, after obtaining a piece of data to be associated from the message queue, it is determined whether the data to be associated obtained in the current cycle contains all the data to be associated: if not, the missing data to be associated is determined, and the target data that has an association relationship with the missing data to be associated is selected from the data to be associated obtained in the current cycle, and the target data is stored in the preset cache. Thereafter, when the missing data to be associated is received, the target data is read from the preset cache, and the missing data to be associated is associated with the target data, thereby achieving the correct association of multiple preset types of data to be associated, solving the problem of being unable to be associated due to the arrival of the data to be associated one after another. The above solution does not require the use of a sliding window, does not increase the amount of calculation, and does not affect the calculation accuracy.

[0046] The first embodiment of the present invention is described below. Figure 2 and Figure 3 The first embodiment takes the association of order data and waybill data as an example to illustrate. Order data and waybill data are two preset types, which are respectively in two topics of the message queue (KAFKA). The consumer (Spark-Streaming) consumes messages from the message queue to obtain data for association. The association result data formed by the association can be sent to another independent topic of the message queue for transmission.

[0047] In the first embodiment, order data a and three waybill data b, c, d form a group of data to be associated. In the first case, if the consumer only receives order data a in the first cycle and does not receive waybill data b, c, d, order data a can be stored in the cache as the target data; if the consumer receives waybill data b and c in the second cycle, order data a that is associated with waybill data b and c can be queried from the cache (at this time, order data a is specific data), and then the association between order data a and waybill data b and the association between order data a and waybill data c are respectively performed, and the two generated associated result data are sent to the corresponding topic of the message queue. Since waybill data d is currently missing, order data a that is associated with waybill data d in the target data can be kept in the cache as the destination data. If the consumer receives waybill data d in the third cycle, order data a can be queried from the cache and the association between order data a and waybill data d can be performed, and finally the generated associated result data is sent to the corresponding topic of the message queue, thereby achieving the correct association of the data to be associated that arrive successively. For specific execution steps, see Figure 3 .

[0048] In step S201, the consumer obtains the first data to be associated from the message queue; in step S202, after the consumer monitors the expiration of the current cycle, it determines whether the corresponding set of data to be associated has all arrived: if not, execute step S203; if arrived, execute step S215 after executing the association on all the data to be associated. In step S203, the consumer executes the association on the part of the data to be associated that has arrived currently and executes step S215 after the association is completed; in step S204, the consumer determines the missing data to be associated; in step S205, the consumer determines the target data that has an association relationship with the missing data to be associated; in step S206, the consumer stores the target data in the preset cache; in step S207, the consumer receives the missing data to be associated in a later cycle and determines whether the corresponding set of data to be associated has all arrived: if not, execute step S208; if arrived, execute the data association after reading the target data from the cache, and execute step S215 after the association is completed. In step S208, the consumer reads specific data that has an association relationship with the acquired data to be associated from the preset cache; in step S209, the consumer associates the specific data with the acquired data to be associated, and executes step S215 after the association is completed; in step S210, the consumer determines the data to be associated that is still missing; in step S211, the consumer determines the target data that has an association relationship with the still missing data to be associated from the target data stored in the preset cache; in step S212, the consumer deletes the target data except the target data in the preset cache; in step S213, the consumer obtains all the remaining data to be associated; in step S214, the consumer reads the target data from the preset cache to perform the association; in step S215, after the association is completed, the consumer sends the association result data to the independent topic of the message queue to realize the transmission of the association result data.

[0049] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0050] For the above-mentioned method embodiments, for the convenience of description, they are expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the order of the actions described, and some steps can actually be performed in other orders or simultaneously. In addition, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary to implement the present invention.

[0051] In order to better implement the above-mentioned solution of the embodiment of the present invention, relevant devices for implementing the above-mentioned solution are also provided below.

[0052] See also Figure 4 As shown, the data association device 400 provided in the embodiment of the present invention is used to associate multiple pieces of data to be associated that belong to multiple preset types to form at least one piece of association result data; any association result data is formed by multiple pieces of data to be associated with a pre-established association relationship; the device is set at the consumer of the preset message queue, and may include: a judgment unit 401, a cache unit 402 and an association unit 403.

[0053] Among them, the judgment unit 401 is used to judge whether the data to be associated obtained in the current cycle contains all the data to be associated in response to obtaining the data to be associated from the message queue; the cache unit 402 is used to determine the missing data to be associated when it is judged that the data is not included, select the target data that has an association relationship with the missing data to be associated from the data to be associated obtained in the current cycle, and store the target data in the preset cache; the association unit 403 is used to read the target data from the preset cache in response to receiving the missing data to be associated in a later cycle, and associate the missing data to be associated with the target data.

[0054] In the embodiment of the present invention, the associating unit may be further configured to: in response to the expiration of the current period, associate the to-be-associated data having an associative relationship among the to-be-associated data acquired in the current period.

[0055] In practical applications, the association unit 403 can be further used to: in response to receiving all the missing data to be associated in the later period, read the target data from the preset cache, and associate all the missing data to be associated with the target data; after the association is completed, delete all the target data in the preset cache.

[0056] In a specific application, the cache unit 402 can be further used to: determine the currently missing data to be associated in response to receiving part of the missing data to be associated in the later period; the association unit 403 can be further used to: read specific data that has an association relationship with the part of the missing data to be associated from the preset cache, and associate the specific data with the part of the missing data to be associated; retain the target data in the target data that has an association relationship with the currently missing data to be associated in the preset cache, and delete the target data other than the target data from the preset cache.

[0057] Preferably, the message queue adopts a subscription and publishing mode, and the data to be associated of the same preset type is in an independent topic of the message queue; and the device 400 may further include: a result processing unit, which is used to: after obtaining the associated result data, send the associated result data to an independent topic of the message queue to execute the transmission of the associated result data.

[0058] In addition, in the embodiment of the present invention, each preset type includes at least one piece of data to be associated, each piece of associated result data corresponds to the plurality of preset types of data to be associated, and the consumer is a stream computing engine.

[0059] According to the technical solution of the embodiment of the present invention, after obtaining a piece of data to be associated from the message queue, it is determined whether the data to be associated obtained in the current cycle contains all the data to be associated: if not, the missing data to be associated is determined, and the target data that has an association relationship with the missing data to be associated is selected from the data to be associated obtained in the current cycle, and the target data is stored in the preset cache. Thereafter, when the missing data to be associated is received, the target data is read from the preset cache, and the missing data to be associated is associated with the target data, thereby achieving the correct association of multiple preset types of data to be associated, solving the problem of being unable to be associated due to the arrival of the data to be associated one after another. The above solution does not require the use of a sliding window, does not increase the amount of calculation, and does not affect the calculation accuracy.

[0060] Figure 5 An exemplary system architecture 500 is shown to which the data association method or data association device according to the embodiment of the present invention can be applied.

[0061] like Figure 5 As shown, the system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505 (this architecture is only an example, and the components included in the specific architecture may be adjusted according to the specific circumstances of the invention). The network 504 is used to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0062] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various client applications, such as big data processing applications (only as an example), can be installed on terminal devices 501, 502, 503.

[0063] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0064] The server 505 may be a server that provides various services, such as a backend server (only an example) that provides support for big data processing applications operated by users using the terminal devices 501, 502, and 503. The backend server may process the received data association request and feed back the processing result (such as whether the association is successful - only an example) to the terminal devices 501, 502, and 503.

[0065] It should be noted that the data association method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data association device is generally disposed in the server 505 .

[0066] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0067] The present invention also provides an electronic device. The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the data association method provided by the present invention.

[0068] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of an electronic device suitable for implementing an embodiment of the present invention. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0069] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0070] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 606 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0071] In particular, according to the embodiments disclosed in the present invention, the process described in the main step diagram above can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the main step diagram. In the above embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit 601, the above functions defined in the system of the present invention are executed.

[0072] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0073] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0074] The units involved in the embodiments of the present invention may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a judgment unit, a cache unit and an association unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the cache unit may also be described as a "unit that provides target data to the association unit".

[0075] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the device, the steps executed by the device include: in response to obtaining the data to be associated from the message queue, judging whether the data to be associated obtained in the current cycle contains all the data to be associated; if not, determining the missing data to be associated, selecting the target data that has an association relationship with the missing data to be associated from the data to be associated obtained in the current cycle, and storing the target data in the preset cache; in response to receiving the missing data to be associated in the later cycle, reading the target data from the preset cache, and associating the missing data to be associated with the target data.

[0076] In the technical solution of the embodiment of the present invention, after obtaining a piece of data to be associated from the message queue, it is determined whether the data to be associated obtained in the current cycle contains all the data to be associated: if not, the missing data to be associated is determined, and the target data that has an association relationship with the missing data to be associated is selected from the data to be associated obtained in the current cycle, and the target data is stored in the preset cache. Thereafter, when the missing data to be associated is received, the target data is read from the preset cache, and the missing data to be associated is associated with the target data, thereby achieving the correct association of multiple preset types of data to be associated, solving the problem of being unable to be associated due to the arrival of the data to be associated one after another. The above solution does not require the use of a sliding window, does not increase the amount of calculation, and does not affect the calculation accuracy.

[0077] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data association method, characterized in that: Used to associate multiple pieces of to-be-associated data belonging to multiple preset types to form at least one piece of associated result data; Any association result data is formed by a plurality of pieces of to-be-associated data with pre-established association relationships; The method is executed by a consumer of a preset message queue, and includes: In response to obtaining the data to be associated from the message queue, determining whether the data to be associated obtained in the current cycle includes all the data to be associated; If not included, determining the missing data to be associated, selecting target data having an associated relationship with the missing data to be associated from the data to be associated acquired in the current cycle, and storing the target data in a preset cache; In response to receiving the missing data to be associated in a later period, the target data is read from the preset cache, and the missing data to be associated is associated with the target data.

2. The method according to claim 1, characterized in that The method further comprises: In response to the current cycle expiring, associating the to-be-associated data with an associative relationship among the to-be-associated data acquired in the current cycle.

3. The method according to claim 1, characterized in that In response to receiving the missing data to be associated in the later period, reading the target data from the preset cache, and associating the missing data to be associated with the target data, comprises: In response to receiving all of the missing data to be associated in the later period, reading the target data from the preset cache, and associating all of the missing data to be associated with the target data; The method further comprises: after the association is completed, deleting all target data in the preset cache.

4. The method according to claim 1, characterized in that: The method further comprises: In response to receiving part of the missing data to be associated in the later period, determining currently missing data to be associated; Reading specific data associated with the portion of the missing data to be associated from the preset cache, and associating the specific data with the portion of the missing data to be associated; The target data in the target data that is associated with the currently missing data to be associated is retained in the preset cache, and the target data other than the target data is deleted from the preset cache.

5. The method according to claim 1, characterized in that The message queue adopts a subscription publishing mode, and the to-be-associated data of the same preset type is in an independent topic of the message queue; and the method further comprises: After obtaining the correlation result data, the correlation result data is sent to an independent topic of the message queue to perform transmission of the correlation result data.

6. The method according to claim 1, characterized in that Each preset type includes at least one piece of data to be associated, each piece of associated result data corresponds to the plurality of preset types of data to be associated, and the consumer is a stream computing engine.

7. A data association device, characterized in that: Used to associate multiple pieces of to-be-associated data belonging to multiple preset types to form at least one piece of associated result data; Any association result data is formed by a plurality of pieces of to-be-associated data with pre-established association relationships; The device is provided at a consumer of a preset message queue, including: a judgment unit, configured to, in response to obtaining the data to be associated from the message queue, judge whether the data to be associated obtained in the current cycle includes all the data to be associated; a cache unit, configured to determine the missing data to be associated if it is determined that the data is not included, select target data having an associated relationship with the missing data to be associated from the data to be associated acquired in the current cycle, and store the target data in a preset cache; The association unit is configured to, in response to receiving the missing data to be associated in a later period, read the target data from the preset cache and associate the missing data to be associated with the target data.

8. The device according to claim 7, characterized in that The associating unit is further configured to: in response to the expiration of the current cycle, associate the to-be-associated data having an associative relationship among the to-be-associated data acquired in the current cycle.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images