Multi-data stream merging method and device, equipment and computer storage medium

By performing partition consumption and parallel processing on multiple data streams, and confirming abnormal data in combination with preset filtering rules, the problem of high data error rate caused by improper processing of data stream correspondence in multiple data stream merging is solved, and high-accurate data merging is achieved .

CN118964412APending Publication Date: 2024-11-15QINGDAO HAIER WASHING MASCH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310547440.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The prior art fails to effectively process the correspondence between data streams when merging multiple data streams, resulting in data arrival time disorder and erroneous merged data streams, which are difficult to use in data mining.

Method used

By partitioning the data in the data stream, using message queues to ensure the correspondence between the data streams, using parallel consumption and preset filtering rules to confirm the abnormal data information, eliminating the abnormal data, and ensuring the accuracy of the merged data stream.

Benefits of technology

It effectively avoids the order of merged data flow caused by the disorder of data arrival time, improves the accuracy of merged data flow, and ensures the reliability of data mining.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964412A_ABST
    Figure CN118964412A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a multi-data-stream merging method and device, equipment and a computer storage medium. Acquiring data information in at least two data streams and labels of the data information, wherein the labels are used for indicating types of the data information; the data information containing the same label is stored in the same partition, the data information belonging to the same data flow in the same partition is stored in the same message queue, and the same partition comprises at least two message queues; performing parallel consumption on different message queues in the same partition to obtain data information after consumption; and merging the consumed data information to obtain a merged data stream. According to the embodiment of the invention, the data information in each data stream is partitioned to ensure the corresponding relationship between the data streams, and the data for merging is confirmed in a message queue consumption mode, so that the situation that the data stream merging sequence is disordered due to disordered data arrival time is avoided, and the accuracy of the merged data is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a method, device, equipment and computer storage medium for merging multiple data streams. Background Art

[0002] In the era of the Internet of Everything, smart devices are increasingly used in people's daily lives. Smart device providers receive massive amounts of real-time data generated by smart devices every day. In order to better mine and utilize these real-time data, they first need to merge these real-time data.

[0003] When merging multiple data streams in the prior art, a common method is to record the time when the data in each data stream arrives at the server, sort the collected data in chronological order, and thus directly merge the multiple data streams.

[0004] However, when merging multiple data streams, it is important to pay attention to the correspondence between the multiple data streams. The above method directly merges all data streams without considering the data backlog in some data streams, which causes the data arrival time to be disordered. The wrong arrival time will destroy the correspondence between the multiple data streams, resulting in a high error rate in the final merged data stream, which is difficult to be used for data mining. Summary of the invention

[0005] The present application provides a method, device, equipment and computer storage medium for merging multiple data streams, which ensures the correspondence between data streams and the accuracy of the merged data by partitioning the data in each data stream.

[0006] In a first aspect, the present application provides a method for merging multiple data streams, comprising:

[0007] Acquire data information and labels of the data information in at least two data streams, where the labels are used to indicate types of the data information;

[0008] The data information with the same tag is stored in the same partition, and the data information belonging to the same data stream in the same partition is stored in the same message queue, wherein the same partition includes at least two message queues;

[0009] Consume different message queues in the same partition in parallel to obtain the data information after consumption;

[0010] The consumed data information is merged to obtain a merged data stream.

[0011] In one possible implementation, the data stream includes a device data stream and an application data stream;

[0012] The same partition includes a device message queue corresponding to the device data flow and an application message queue corresponding to the application data flow.

[0013] In a possible implementation manner, the data information includes device data information and application data information, and both the device data information and the application data information include a trigger time;

[0014] The parallel consumption of different message queues in the same partition includes:

[0015] Determining whether the device message queue and the application message queue meet the consumption conditions according to the triggering time of the current device data information and the triggering time of the current application data information;

[0016] Consume the message queues that meet the consumption conditions.

[0017] In a possible implementation, judging whether the device message queue and the application message queue meet the consumption condition according to the triggering time of the current device data information and the triggering time of the current application data information includes:

[0018] Determine whether the triggering time of the current device data information is less than the triggering time of the current application data information;

[0019] If so, the device message queue meets the consumption condition, and the application data queue does not meet the consumption condition;

[0020] If not, the device message queue meets the consumption condition and the application data queue meets the consumption condition.

[0021] In a possible implementation, merging the consumed data information includes:

[0022] In the consumed data information, confirming abnormal data information according to preset screening rules;

[0023] The abnormal data information is eliminated, and the remaining consumed data information is merged.

[0024] In a possible implementation manner, both the device data information and the application data information include a service type, and the confirming the abnormal data information according to a preset screening rule includes:

[0025] Using the triggering time of the application data information as the first triggering time, acquiring device data information having the same service type as the application data information, and using the triggering time of the device data information as the second triggering time;

[0026] When the second trigger time is greater than the first trigger time, determining whether the difference between the second trigger time and the first trigger time is greater than a threshold of the service type;

[0027] If so, confirm that the application data information is abnormal data information.

[0028] In a possible implementation, merging the consumed data information includes:

[0029] Store the consumed data information into the data subject area corresponding to each partition to obtain the partition data stream;

[0030] Store the partitioned data stream into the total data subject area to obtain the final merged data stream.

[0031] In a second aspect, the present application provides a device for merging multiple data streams, the device for merging multiple data streams comprising a data stream input module, a partitioning module, a consumption module and a merging module, wherein:

[0032] A data stream input module, used to obtain data information in at least two data streams, wherein the data information includes the data stream to which it belongs and at least one label;

[0033] A partitioning module, used for partitioning the data information according to the label, and storing the data information with consistent data streams in the same partition into the same message queue, wherein the same partition includes at least two message queues;

[0034] The consumption module is used to consume different message queues in the same partition in parallel and obtain the data information after consumption;

[0035] The merging module is used to merge the consumed data information to obtain a merged data stream.

[0036] In a possible implementation, the data stream acquired by the data stream input module specifically includes a device data stream and an application data stream; in the partition module, the same partition includes a device message queue corresponding to the device data stream and an application message queue corresponding to the application data stream.

[0037] In a possible implementation, the data information includes device data information and application data information, the device data information and the application data information both include a trigger time, and the consumption module is specifically used to:

[0038] Determining whether the device message queue and the application message queue meet the consumption conditions according to the triggering time of the current device data information and the triggering time of the current application data information;

[0039] Consume the message queues that meet the consumption conditions.

[0040] In a possible implementation manner, the consumption module is specifically used for:

[0041] Determine whether the triggering time of the current device data information is less than the triggering time of the current application data information;

[0042] If so, the device message queue meets the consumption condition, and the application data queue does not meet the consumption condition;

[0043] If not, the device message queue meets the consumption condition and the application data queue meets the consumption condition.

[0044] In a possible implementation, the merging module is specifically used for:

[0045] In the consumed data information, confirm abnormal data information according to preset screening rules;

[0046] The abnormal data information is eliminated, and the remaining consumed data information is merged.

[0047] In a possible implementation manner, the device data information and the application data information both include a service type, and the merging module is specifically configured to:

[0048] Using the triggering time of the application data information as the first triggering time, acquiring device data information having the same service type as the application data information, and using the triggering time of the device data information as the second triggering time;

[0049] When the second trigger time is greater than the first trigger time, determining whether the difference between the second trigger time and the first trigger time is greater than a threshold of the service type;

[0050] If so, confirm that the application data information is abnormal data information.

[0051] In a possible implementation manner, the merging module is further used for:

[0052] Store the consumed data information into the data subject area corresponding to each partition to obtain the partition data stream;

[0053] Store the partitioned data stream into the total data subject area to obtain the final merged data stream.

[0054] In a third aspect, an embodiment of the present application provides a device for merging multiple data streams, including: a processor and a memory;

[0055] The memory stores computer-executable instructions;

[0056] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method for merging multiple data streams as described in any one of the first aspects.

[0057] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored, and the computer program is used to implement the multi-data stream merging method as described in any one of the first aspects.

[0058] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multi-data stream merging method as described in any one of the above items.

[0059] The present application provides a method, apparatus, device and storage medium for merging multiple data streams, which obtain data information and labels of the data information in at least two data streams, wherein the labels are used to indicate the type of the data information, store data information with the same labels in the same partition, store data information belonging to the same data stream in the same partition in the same message queue, wherein the same partition includes at least two message queues to ensure the correspondence between the data streams; consume different message queues in the same partition in parallel to obtain the consumed data information, merge the consumed data information to obtain a merged data stream, confirm the data for merging by means of message queues, avoid disorder in the order of the merged data stream due to disorder in the arrival time of the data, and ensure the accuracy of the data after merging. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] Figure 1 A schematic diagram of the architecture of a system for merging multiple data streams provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of a method for merging multiple data streams provided in an embodiment of the present application;

[0063] Figure 3 A flowchart of a method for confirming abnormal data information according to preset screening rules provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of a multi-data stream merging device provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the structure of an electronic device based on a multi-data stream merging device provided in an embodiment of the present application.

[0066] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein, for example.

[0069] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0070] First, the terms involved in this application are explained:

[0071] A data stream is an ordered sequence of bytes with a start and end point. The transmission of a data stream is a continuous process. Compared with the traditional data transmission process of sending data records in batches, a data stream has a time sequence, so the data in the data stream can be analyzed in real time.

[0072] Kafka: Kafka is a distributed, partitioned, multi-copy distributed messaging system. Its biggest feature is that it can process large amounts of data in real time to meet various demand scenarios.

[0073] In order to better understand the solution of the embodiment of the present application, an application environment involved in the embodiment of the present application is first introduced below.

[0074] See also Figure 1 , Figure 1 The schematic diagram of the architecture of the system for merging multiple data streams provided in the embodiment of the present application is as follows. Figure 1 As shown, the system provided in this embodiment includes several device terminals 100 , several application terminals 200 and a server 300 .

[0075] Among them, the device terminal 100 can send its own status and the status of executing business to the server 300 in the form of data stream. The application terminal 200 can send instructions to the server 300 in the form of data stream to control the server 300 and the device terminal 100 to perform related tasks (such as storing current settings, or controlling the device to turn on, turn off, execute business, etc.), and the server 300 can store and process the data uploaded by the device terminal 100 and the application terminal 200. It can be understood that the device terminal 100 may include smart home appliances, such as smart air conditioners, smart washing machines, smart sweeping robots, etc., the application terminal 200 may include a personal computer, a tablet computer, a smart panel, etc., and the server 300 may include a distributed database, a data warehouse tool, a distributed publish-subscribe message system, etc., and the embodiments of the present application are not limited here.

[0076] It is understandable that with the increasing number of smart devices in people's daily lives, the data sent by the corresponding device terminals 100 and application terminals 200 are growing at a geometric level. The server 300 is receiving the data streams sent by the device terminals 100 and application terminals 200 at all times. In order to better mine and utilize these data, it is first necessary to merge the data received by the server 300.

[0077] The commonly used means of data consolidation is usually to record the time when the server 300 receives the data sent by the device terminal 100 and the application terminal 200, and sort and merge all the data in chronological order. This method can quickly merge all data streams, but it is too general, and the merged data stream obtained is relatively rough and prone to errors. For example, when the network is unstable or the terminal is stuck, the data streams sent by the device terminal 100 and the application terminal 200 are likely to be backlogged, causing the time for the data to arrive at the server 300 to be disordered, and the merged data stream obtained by direct merging will have an incorrect order.

[0078] In particular, for the system for merging multiple data streams provided in this embodiment, there is a certain correspondence between the data streams. For example, if the user controls the smart air conditioner to turn on through the mobile phone application software, the server 300 first receives the data from the mobile phone application software, and then receives the data from the smart air conditioner. Therefore, once the source of the data stream has a data backlog problem, the direct merging method will also destroy the correspondence between the data stream of the device terminal 100 and the data stream of the application terminal 200, resulting in a high error rate of the final merged data stream, which is difficult to be used for data mining.

[0079] In order to solve this technical problem, this embodiment provides a method for merging multiple data streams. The method first partitions the data information in each data stream to ensure the correspondence between the data streams; then confirms the data for merging through a message queue to ensure the accuracy of the data after merging.

[0080] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0081] Figure 2 This is a flowchart of the method for merging multiple data streams provided in the embodiment of the present application. Figure 2 , the method may include:

[0082] S201: Acquire data information and labels of the data information in at least two data streams, where the labels are used to indicate types of the data information.

[0083] The execution subject of the embodiment of the present application may be a server, or a multi-data stream merging device in the server, wherein the multi-data stream merging device may be implemented by software.

[0084] The server can perform network communication through a network interface, thereby receiving at least two data streams. The data stream can originate from a device and / or an application client. It is understandable that the data in the data stream carries data information and a label for indicating the type of the data information. Exemplarily, the label can be a code of the data information, and the data information can be distinguished according to different codes.

[0085] It should be noted that different data streams may come from the same device or client, or from different devices or clients. As an example and not a limitation, there may be multiple user accounts on the same client, and different user accounts send different data streams. The same data source may come from the same client or from different clients. For example, the same user using the same application software account can send data streams on multiple clients. This embodiment does not limit the classification of data streams.

[0086] S202: storing data information with the same tag into the same partition, and storing data information belonging to the same data flow in the same partition into the same message queue, wherein the same partition includes at least two message queues.

[0087] In this step, the partition may be a partition of a distributed messaging system. It is understandable that in a distributed messaging system, a partition is the smallest storage unit, and each partition is a time-ordered set. Therefore, storing data information in a partition can ensure the orderliness of the data information. Optionally, the partition of the distributed messaging system may be Kafka.

[0088] The data information of different data streams in the partition is distinguished to maintain the corresponding relationship between the data streams, and the data information belonging to the same data stream can be stored in the same message queue.

[0089] S203: Consume different message queues in the same partition in parallel to obtain data information after consumption.

[0090] In this step, the process of consuming by the message queue can be to send data information to the consumer, and the consumer obtains the data information after consumption. Each message queue can only consume serially, and the consumption of different consumer queues does not affect each other, so different message queues can consume in parallel. Among them, the consumer refers to the client for receiving and reading data information from the message queue. In the embodiment of the present application, the consumer can also be a data subject area.

[0091] S204: Merge the consumed data information to obtain a merged data stream.

[0092] In this step, all the consumed data information is obtained by the consumer. According to the time when the consumer receives the consumed data information, the consumed data information can be sorted in time sequence to obtain a merged data stream.

[0093] The multi-data stream merging method provided in the embodiment of the present application obtains data information and labels of the data information in at least two data streams, wherein the labels are used to indicate the type of the data information; stores data information with the same labels in the same partition, and stores data information belonging to the same data stream in the same partition in the same message queue, wherein the same partition includes at least two message queues; performs parallel consumption on different message queues in the same partition to obtain data information after consumption; and merges the data information after consumption to obtain a merged data stream. The data information in each data stream is partitioned to ensure the correspondence between the data streams, and the data to be merged is confirmed by consuming the message queue, thereby avoiding disorder in the order of the merged data stream due to disorder in the arrival time of the data, and ensuring the accuracy of the data after the merge.

[0094] The technical solution of the above-mentioned multi-data stream merging method is introduced in detail below.

[0095] In one embodiment, the data stream includes a device data stream and an application data stream. The device data stream may come from a smart home appliance, such as a smart refrigerator, a smart air conditioner, a smart washing machine, etc., which is not limited in this embodiment. The application data stream may come from an application software corresponding to the smart home appliance, and the application software may be a third-party application on a smart mobile device.

[0096] In this embodiment, the label may be a device number, a device model, or a device type. As an example but not a limitation, the label may be a device number, and the data information in the device data stream carries the respective device number information. For example, if the label of the data information in the device data stream is "01", it indicates that the data information is sent by the device numbered 01. The data information in the application data stream also carries the corresponding device number information. For example, if the label of the data information in the application data stream is "01", it indicates that the application software has sent the data information acting on the device numbered 01.

[0097] After storing data with the same tag in the same partition, the data information belonging to the same data flow in the same partition is stored in the same consumption queue. It should be noted that the same partition includes a device consumption queue corresponding to the device data flow and an application consumption queue corresponding to the application data flow.

[0098] In this embodiment, the correspondence between the device data flow and the application data flow is ensured by storing the device data information and the application data information with the same label in the same partition and storing the data information corresponding to different data flows through different consumption queues.

[0099] In one embodiment, the data information includes device data information and application data information, and the device data information and the application data information both include a trigger time. Specifically, the trigger time refers to the time when the device data information is sent from the smart home appliance end and the time when the application data information is sent from the application software end.

[0100] In this embodiment, the parallel consumption of different consumption queues in the same partition includes: judging whether the device message queue and the application consumption queue meet the consumption conditions according to the triggering time of the current device data information and the triggering time of the current application data information; and consuming the message queues that meet the consumption conditions.

[0101] Generally speaking, in the same time period, the device data information is far more than the application data information, which results in more data information in the device message queue and less data information in the application consumption queue. If the data is transmitted normally, the device consumption queue and the application consumption queue can consume at the same time, but if the data source has a backlog of data, the application consumption queue may not have data information for a long time, and a large amount of accumulated data information may suddenly flow in at a certain moment, while the device consumption queue has been consuming continuously. In this case, the data information in the device consumption queue and the data information in the application consumption queue are not equal, and the corresponding relationship is not obvious, which may make the subsequent data mining accuracy low. Therefore, when different consumption queues are consuming in parallel, it is necessary to first determine whether the consumption queue meets the consumption conditions.

[0102] Specifically, judging whether the device message queue and the application consumption queue meet the consumption condition according to the triggering time of the current device data information and the triggering time of the current application data information includes:

[0103] Determine whether the triggering time of the current device data information is less than the triggering time of the current application data information;

[0104] If so, the device message queue meets the consumption condition, and the application data queue does not meet the consumption condition;

[0105] If not, the device message queue meets the consumption condition and the application data queue meets the consumption condition.

[0106] It is understandable that in the normal consumption process of device data flow and application data flow, the triggering time of the current device data information should not be less than the triggering time of the current application data information, that is, if it is the same triggering time, the device data information should not be consumed earlier than the application data information.

[0107] In this embodiment, by judging whether a message queue meets the consumption conditions and consuming the message queue that meets the conditions, the correspondence between data streams can be further guaranteed, and the unequal time sequence of data information in the device consumption queue and the data information in the application consumption queue can be avoided.

[0108] In one embodiment, the merging of the post-consumption data information includes: confirming abnormal data information in the post-consumption data information according to a preset screening rule; removing the abnormal data information, and merging the remaining post-consumption data information.

[0109] In this embodiment, abnormal data information can be application data information that has no corresponding relationship. For example, when the application software end sends an application data information through interaction with the user, but due to network delay, the application data information is received by the server after a long time, and the corresponding device end does not respond to the application data information. Therefore, after obtaining the data information after consumption, it is necessary to filter out the abnormal data information and eliminate the abnormal data information to avoid the randomness of the data.

[0110] It should be noted that both the device data information and the application data information include a service type, which refers to a specific service executed and sent by the device end or a specific service sent by the application software end, such as "startup", "self-cleaning", "water injection", etc.

[0111] Specifically, see Figure 3 , Figure 3 A flowchart of a method for confirming abnormal data information according to preset screening rules provided in this embodiment, Figure 3 The above process of screening out abnormal data information and removing the abnormal data information is described in detail. Figure 3 As shown, the method includes:

[0112] S301: Taking the triggering time of the application data information as the first triggering time, acquiring device data information having the same service type as the application data information, and taking the triggering time of the device data information as the second triggering time.

[0113] It is understandable that when device data information is sent by a smart home appliance, the smart home appliance can be manually operated or controlled by application software, so not every device data information has corresponding application data information, but most of the time, application data information has corresponding device data information. Therefore, in this embodiment, in the post-consumption data, application data information is used as the screening object.

[0114] Specifically, the service type and triggering time of an application data information can be read from the application data information, and the triggering time of the application data information is used as the first triggering time. In the consumed data information, the device data information with the same service type as the application data information is selected, that is, this may be a pair of application data information and device data information with a corresponding relationship. The triggering time of the device data information is read as the second triggering time.

[0115] S302: When the second trigger time is greater than the first trigger time, determine whether the difference between the second trigger time and the first trigger time is greater than a threshold of the service type.

[0116] It is understandable that if the second trigger moment is not greater than the first trigger moment, it indicates that the device end sends the device data information first, and the application software end sends the application device data at the same time or later. This situation may be caused by the user's mistaken touch, or it may be caused by the application software end receiving feedback from the device end, and the application data information and the device data information can be directly merged. However, if the second trigger moment is greater than the first trigger moment, by calculating the difference between the first trigger moment and the second trigger moment, the sending time difference between the application data information and the device data information of the same service type can be obtained, and then it can be determined whether this corresponding relationship is established.

[0117] It should be noted that different thresholds of preset screening rules may be used for different business types. For example, for a washing machine, if the business type is "start", the corresponding threshold may be 10 minutes; if the business type is "self-cleaning", the corresponding threshold may be 20 minutes.

[0118] S303: If yes, confirm that the application data information is abnormal data information.

[0119] When the difference between the second trigger moment and the first trigger moment is greater than the threshold of the service type, it means that the device end has not sent device data information corresponding to the application data information during the period from the second trigger moment to the first trigger moment, that is, within the specified time period, after the application software end sent an application data information pointing to a specific device end and having a clear service type, the corresponding device end did not respond. Therefore, this application data information is confirmed as abnormal data information.

[0120] It can be understood that when the sending time difference between the application data information and the device data information of the same service type meets the threshold, there is a corresponding relationship between the application data information and the device data information.

[0121] In this embodiment, by screening and eliminating abnormal data information of the consumed data information, the contingency caused by the abnormal data information can be avoided, and the accuracy of the data information in the merged data stream can be effectively guaranteed.

[0122] In one embodiment, the merging of the consumed data information further includes: storing the consumed data information into the data subject area corresponding to each partition to obtain a partition data stream; storing the partition data stream into the total data subject area to obtain a final merged data stream.

[0123] In this embodiment, the data subject area may be a topic corresponding to Kafka, where a topic is a message stream belonging to a specific category and can be used to store data.

[0124] It is understandable that the consumed data information is obtained from each partition. In order to better use this data information for data mining, the consumed data information obtained from each partition is first stored in the data subject area corresponding to each partition. Since the data subject area is also a message flow, it can maintain time order, so the partition data stream can be obtained in each data subject area. Similarly, storing the partition data stream in the total data subject area is equivalent to a summary of all partition data streams to obtain the final merged data stream.

[0125] In this embodiment, the post-consumption data information of each partition is first merged to obtain a partition data stream, and then the partition data streams are merged into a final merged data stream. The use of such a two-stage merging method can further refine the corresponding relationship between the data information, which is conducive to the application of the data information as data mining.

[0126] Figure 4 Schematic diagram of a device for merging multiple data streams provided by an embodiment of the present invention. Figure 4 As shown, the multi-data stream merging device includes:

[0127] The data stream input module 41 is used to obtain data information in at least two data streams, where the data information includes the data stream to which it belongs and at least one label.

[0128] The partition module 42 is used to partition the data information according to the label, and store the data information belonging to the same data stream in the same partition into the same message queue, and the same partition includes at least two message queues.

[0129] The consumption module 43 is used to perform parallel consumption on different message queues in the same partition to obtain data information after consumption.

[0130] The merging module 44 is used to merge the consumed data information to obtain a merged data stream.

[0131] In one possible design, the data stream acquired by the data stream input module 41 specifically includes a device data stream and an application data stream; in the partition module, the same partition includes a device message queue corresponding to the device data stream and an application message queue corresponding to the application data stream.

[0132] In a possible design, the data information includes device data information and application data information, the device data information and the application data information both include a trigger time, and the consumption module 43 is specifically used to:

[0133] Determining whether the device message queue and the application message queue meet the consumption conditions according to the triggering time of the current device data information and the triggering time of the current application data information;

[0134] Consume the message queues that meet the consumption conditions.

[0135] In a possible design, the consumption module 43 is specifically used for:

[0136] Determine whether the triggering time of the current device data information is less than the triggering time of the current application data information;

[0137] If so, the device message queue meets the consumption condition, and the application data queue does not meet the consumption condition;

[0138] If not, the device message queue meets the consumption condition and the application data queue meets the consumption condition.

[0139] In a possible design, the merging module 44 is specifically used for:

[0140] In the consumed data information, confirm abnormal data information according to preset screening rules;

[0141] The abnormal data information is eliminated, and the remaining consumed data information is merged.

[0142] In a possible design, the device data information and the application data information both include a service type, and the merging module 44 is specifically configured to:

[0143] Using the triggering time of the application data information as the first triggering time, acquiring device data information having the same service type as the application data information, and using the triggering time of the device data information as the second triggering time;

[0144] When the second trigger time is greater than the first trigger time, determining whether the difference between the second trigger time and the first trigger time is greater than a threshold of the service type;

[0145] If so, confirm that the application data information is abnormal data information.

[0146] In a possible design, the merging module 44 is further used for:

[0147] Store the consumed data information into the data subject area corresponding to each partition to obtain the partition data stream;

[0148] Store the partitioned data stream into the total data subject area to obtain the final merged data stream.

[0149] The multi-data stream merging device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0150] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0151] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0152] Figure 5 This is a schematic diagram of the structure of an electronic device based on a multi-data stream merging device provided in an embodiment of the present application. Figure 5 As shown, the electronic device of this embodiment includes: at least one processor 50 ( Figure 5 Only one is shown in the figure) a processor, a memory 51, and a computer program stored in the memory 51 and executable on at least one processor 50. When the processor 50 executes the computer program, the steps in any of the above-mentioned cloud platform tenant modeling test method embodiments are implemented.

[0153] The electronic device may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5 These are merely examples of electronic devices and do not constitute a limitation on the electronic devices. The electronic devices may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, the electronic devices may also include input and output devices, network access devices, etc.

[0154] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0155] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0156] In some embodiments, the memory 51 may be an internal storage unit of the electronic device, such as the memory of the electronic device. In other embodiments, the memory 51 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 51 may also include both an internal storage unit of the electronic device and an external storage device. The memory 51 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program. The memory 51 may also be used to temporarily store data that has been output or is to be output.

[0157] The embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0158] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0159] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the above-mentioned electronic device.

[0160] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0161] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0162] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] So far, the technical solution of the present application has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments, and the above embodiments are only used to illustrate the technical solution of the present application rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for merging multiple data streams, It is characterized in that The method comprises: Acquire data information and labels of the data information in at least two data streams, where the labels are used to indicate types of the data information; The data information with the same tag is stored in the same partition, and the data information belonging to the same data stream in the same partition is stored in the same message queue, wherein the same partition includes at least two message queues; Consume different message queues in the same partition in parallel to obtain the data information after consumption; The consumed data information is merged to obtain a merged data stream.

2. The method according to claim 1, It is characterized in that The data stream includes a device data stream and an application data stream; The same partition includes a device message queue corresponding to the device data flow and an application message queue corresponding to the application data flow.

3. The method according to claim 2, It is characterized in that The data information includes device data information and application data information, and both the device data information and the application data information include a trigger time; The parallel consumption of different message queues in the same partition includes: Determining whether the device message queue and the application message queue meet the consumption conditions according to the triggering time of the current device data information and the triggering time of the current application data information; Consume the message queues that meet the consumption conditions.

4. The method according to claim 3, It is characterized in that The determining whether the device message queue and the application message queue meet the consumption condition according to the triggering time of the current device data information and the triggering time of the current application data information includes: Determine whether the triggering time of the current device data information is less than the triggering time of the current application data information; If so, the device message queue meets the consumption condition, and the application data queue does not meet the consumption condition; If not, the device message queue meets the consumption condition and the application data queue meets the consumption condition.

5. The method according to claim 3, It is characterized in that The merging of the consumed data information includes: In the consumed data information, confirm abnormal data information according to preset screening rules; The abnormal data information is eliminated, and the remaining consumed data information is merged.

6. The method according to claim 5, It is characterized in that The device data information and the application data information both include a business type, and the confirming of abnormal data information according to a preset screening rule includes: Using the triggering time of the application data information as the first triggering time, acquiring device data information having the same service type as the application data information, and using the triggering time of the device data information as the second triggering time; When the second trigger time is greater than the first trigger time, determining whether the difference between the second trigger time and the first trigger time is greater than a threshold of the service type; If so, confirm that the application data information is abnormal data information.

7. The method according to claim 1, It is characterized in that The merging of the consumed data information includes: Store the consumed data information into the data subject area corresponding to each partition to obtain the partition data stream; Store the partitioned data stream into the total data subject area to obtain the final merged data stream.

8. A device for merging multiple data streams, include: A data stream input module, used to obtain data information in at least two data streams, wherein the data information includes the data stream to which it belongs and at least one label; A partitioning module, used for partitioning the data information according to the label, and storing the data information with consistent data streams in the same partition into the same message queue, wherein the same partition includes at least two message queues; The consumption module is used to consume different message queues in the same partition in parallel and obtain the data information after consumption; The merging module is used to merge the consumed data information to obtain a merged data stream.

9. An electronic device, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method for merging multiple data streams according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method for merging multiple data streams as claimed in any one of claims 1 to 7 when executed by a processor.