Rapid image archiving method and system based on Kafka

Abstract

The invention provides a Kafka-based image rapid archiving method and system. The method comprises the following steps: determining a target partition according to a to-be-processed file and a preset theme; determining a target writing mode according to a to-be-processed file and the target partition; generating a message queue according to the target writing mode and the target partition; determining a target file in the message queue according to a preset user demand; and archiving information is generated according to the target file, and the archiving information comprises an archiving condition, an archiving log and a failure list. The archiving speed is higher, the fault-tolerant mechanism is more perfect, and the process of operation and maintenance and troubleshooting is greatly simplified.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a Kafka-based rapid image archiving method and system. Background Art

[0002] In the past one or two years, the layout of cloud computing in the medical industry has accelerated significantly. At the same time, IT giants have joined hands with medical system integrators to develop and design medical clouds as an important goal. They are also preparing to build medical imaging cloud platforms in various cities and regions to enable their data to be interconnected. However, when medical data is put on the cloud, how to quickly archive image files has become a problem.

[0003] The existing rapid archiving technology is that when writing images to Kafka, there are partition queue hotspots, but there is a queue congestion, which leads to untimely archiving. In addition, the Kafka server disk is easily full. Once the Kafka cluster goes down, the entire service is unavailable, and there is no fault tolerance mechanism. In addition, there is no visual interface, which makes operation and maintenance more difficult, making it difficult to troubleshoot problems and making it impossible for technical personnel to troubleshoot. Summary of the Invention

[0004] In view of the above problems, the present application is proposed to provide a Kafka-based method and system for rapid image archiving that overcomes or at least partially solves the above problems, including:

[0005] A Kafka-based image rapid archiving method, the method comprising:

[0006] Determine the target partition based on the files to be processed and the preset topic;

[0007] Determining a target writing mode according to the file to be processed and the target partition;

[0008] generating a message queue according to the target writing mode and the target partition;

[0009] Determining a target file in the message queue according to a preset user requirement;

[0010] Archiving information is generated according to the target file, wherein the archiving information includes archiving status, archiving log and failure list.

[0011] Furthermore, the step of determining the target partition based on the file to be processed and the preset theme, wherein the target partition includes the first target partition and / or the second target partition, includes:

[0012] determining a target data field in the file to be processed;

[0013] Converting the target data field to generate a byte array;

[0014] Generate a first target partition according to the byte array and the total number of partitions of the current topic; and / or,

[0015] A second target partition is generated according to the target data field, the byte array, and the total number of partitions of the current topic.

[0016] Furthermore, the step of generating the first target partition according to the byte array and the total number of partitions of the current topic includes:

[0017] Generate a hash value based on the byte array and a preset algorithm, where the hash value is a positive number;

[0018] The first target partition is determined according to the total number of partitions of the current topic and the hash value.

[0019] Furthermore, the step of generating a second target partition according to the target data field, the byte array, and the total number of partitions of the current topic includes:

[0020] Determine partition usage according to the total number of partitions of the current topic, the byte array, and the target data field;

[0021] A partition with the smallest data volume is determined according to the partition usage, and the partition with the smallest data volume is marked as the second target partition.

[0022] Furthermore, the step of determining the target writing mode based on the file to be processed and the target partition includes:

[0023] Writing the to-be-processed file to the target partition and determining whether there is degradation in the writing process;

[0024] When the degradation situation exists, the target writing mode is determined according to the degradation situation.

[0025] Further, when the degradation situation exists, determining the target writing mode is generated according to the degradation situation, wherein the degradation situation includes the first degradation situation or the second degradation situation, including:

[0026] When the number of times the target image file is sent is greater than a first preset number, it is determined to be the first degradation situation, the target image file is stored in a first preset storage unit, and marked as the target writing mode of the first degradation situation; or

[0027] When the abnormal number of times the available topic list is obtained through the application interface is greater than the preset second preset number, it is determined to be the second degradation situation, the target image file is stored in the second preset storage unit, and marked as the target writing mode of the second degradation situation.

[0028] Furthermore, the step of determining the target file in the message queue according to the preset user demand includes:

[0029] Determining consumption information in the message queue according to preset user requirements;

[0030] The consumption information is analyzed in a preset manner to generate analysis data;

[0031] Generating target query structure information from the parsed data according to index topics and assembly formats corresponding to the index topics;

[0032] A target file is determined in the message queue according to the target query structure information.

[0033] An embodiment of the present application further discloses a Kafka-based rapid image archiving system, the system comprising:

[0034] A first determination module is used to determine a target partition based on the file to be processed and a preset topic;

[0035] A second determining module is used to determine a target writing mode according to the file to be processed and the target partition;

[0036] A first generating module, configured to generate a message queue according to the target writing mode and the target partition;

[0037] A third determining module is used to determine a target file in the message queue according to a preset user requirement;

[0038] The second generating module is used to generate archiving information according to the target file, wherein the archiving information includes archiving status, archiving log and failure list.

[0039] An embodiment of the present application further discloses a computer device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of the Kafka-based rapid image archiving method as described above are implemented.

[0040] An embodiment of the present application further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the Kafka-based rapid image archiving method as described above are implemented.

[0041] This application has the following advantages:

[0042] In the embodiments of the present application, compared with the prior art in which "the existing rapid archiving technology is that when writing images to Kafka, there are partition queue hotspots, but there is a queue congestion, which leads to untimely archiving, and the Kafka server disk is easily full. Once the Kafka cluster goes down, the entire service can be unused, and there is no fault tolerance mechanism; in addition, there is no visual interface, which increases the difficulty of operation and maintenance, makes it difficult to troubleshoot problems, and technical personnel are unable to troubleshoot", the present application provides "a solution that can achieve rapid archiving of image files based on Kafka", specifically: "A method for rapid image archiving based on Kafka, the method comprising: determining a target partition based on the file to be processed and a preset topic; determining a target writing method based on the file to be processed and the target partition; generating a message queue based on the target writing method and the target partition; determining a target file in the message queue based on preset user requirements; generating archiving information based on the target file, wherein the archiving information includes archiving status, archiving logs and failure lists". By "determining the target partition according to the files to be processed and the preset topic; determining the target writing method according to the files to be processed and the target partition; generating a message queue according to the target writing method and the target partition; determining the target file in the message queue according to the preset user needs; generating archiving information according to the target file, wherein the archiving information includes archiving status, archiving log and failure list", the problem of "the existing fast archiving technology is that there is a partition queue hotspot when writing images to Kafka, but there is a queue congestion, resulting in untimely archiving, and the server disk of Kafka is easily full. Once the Kafka cluster goes down, the entire service can be unused, and there is no fault tolerance mechanism; in addition, there is no visual interface, which increases the difficulty of operation and maintenance, makes it difficult to troubleshoot problems, and technical personnel cannot troubleshoot" is achieved. The Kafka consumption process has been optimized, and an enhanced fault-tolerance mechanism has been added to ensure the stability and reliability of data processing. An interface-based management tool has been provided to facilitate monitoring and management of the entire process. When writing images to Kafka based on the original method, by optimizing the partitioning strategy and algorithm for writing images to Kafka, queue congestion (hotspot partitioning problem) has been effectively avoided, and the speed and efficiency of consumption archiving have been significantly improved. The risk of rapid depletion of Kafka disk space has been resolved, and the high availability of the system has been improved and enhanced, thereby ensuring more stable data processing performance. New index topic processing has been added to specifically handle indexes, and image topics are only responsible for image processing. When archiving and consuming Kafka images, visualization has been achieved, allowing users to view the current consumption status, as well as the interface for consumption failure and retry processing, which greatly simplifies the operation and maintenance and troubleshooting process. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 This is a flowchart of a Kafka-based method for rapid image archiving provided by an embodiment of the present application;

[0045] Figure 2 This is a structural block diagram of a Kafka-based image rapid archiving system provided by an embodiment of the present application;

[0046] Figure 3 This is a schematic structural diagram of a computer device provided by one embodiment of the present invention;

[0047] Figure 4 It is a flowchart of an embodiment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the objectives, features, and advantages of this application more readily apparent, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It is apparent that the embodiments described are only a portion of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments in this application without inventive effort are also within the scope of protection of this application.

[0049] Reference Figure 1 and Figure 4 , showing a step flow chart and a flow diagram of a Kafka-based image rapid archiving method provided by an embodiment of the present application;

[0050] A Kafka-based image rapid archiving method, the method comprising:

[0051] S110, determining a target partition based on the file to be processed and a preset theme;

[0052] S120, determining a target writing mode according to the file to be processed and the target partition;

[0053] S130, generating a message queue according to the target writing mode and the target partition;

[0054] S140, determining a target file in the message queue according to a preset user requirement;

[0055] S150: Generate archiving information based on the target file, wherein the archiving information includes archiving status, archiving log, and failure list.

[0056] In the embodiments of the present application, compared with the prior art in which "the existing rapid archiving technology is that when writing images to Kafka, there are partition queue hotspots, but there is a queue congestion, which leads to untimely archiving, and the Kafka server disk is easily full. Once the Kafka cluster goes down, the entire service can be unused, and there is no fault tolerance mechanism; in addition, there is no visual interface, which increases the difficulty of operation and maintenance, makes it difficult to troubleshoot problems, and technical personnel are unable to troubleshoot", the present application provides "a solution that can achieve rapid archiving of image files based on Kafka", specifically: "A method for rapid image archiving based on Kafka, the method comprising: determining a target partition based on the file to be processed and a preset topic; determining a target writing method based on the file to be processed and the target partition; generating a message queue based on the target writing method and the target partition; determining a target file in the message queue based on preset user requirements; generating archiving information based on the target file, wherein the archiving information includes archiving status, archiving logs and failure lists". By "determining the target partition according to the files to be processed and the preset topic; determining the target writing method according to the files to be processed and the target partition; generating a message queue according to the target writing method and the target partition; determining the target file in the message queue according to the preset user needs; generating archiving information according to the target file, wherein the archiving information includes archiving status, archiving log and failure list", the problem of "the existing fast archiving technology is that there is a partition queue hotspot when writing images to Kafka, but there is a queue congestion, resulting in untimely archiving, and the server disk of Kafka is easily full. Once the Kafka cluster goes down, the entire service can be unused, and there is no fault tolerance mechanism; in addition, there is no visual interface, which increases the difficulty of operation and maintenance, makes it difficult to troubleshoot problems, and technical personnel cannot troubleshoot" is achieved. The Kafka consumption process has been optimized, and an enhanced fault-tolerance mechanism has been added to ensure the stability and reliability of data processing. An interface-based management tool has been provided to facilitate monitoring and management of the entire process. When writing images to Kafka based on the original method, by optimizing the partitioning strategy and algorithm for writing images to Kafka, queue congestion (hotspot partitioning problem) has been effectively avoided, and the speed and efficiency of consumption archiving have been significantly improved. The risk of rapid depletion of Kafka disk space has been resolved, and the high availability of the system has been improved and enhanced, thereby ensuring more stable data processing performance. New index topic processing has been added to specifically handle indexes, and image topics are only responsible for image processing. When archiving and consuming Kafka images, visualization has been achieved, allowing users to view the current consumption status, as well as the interface for consumption failure and retry processing, which greatly simplifies the operation and maintenance and troubleshooting process.

[0057] Next, a Kafka-based rapid image archiving method in this exemplary embodiment will be further described.

[0058] As described in step S110 , the target partition is determined based on the file to be processed and the preset theme.

[0059] It should be noted that the target partition includes a first target partition and / or a second target partition.

[0060] In one embodiment of the present invention, the specific process of "determining the target partition according to the file to be processed and the preset theme" in step S110 can be further explained in combination with the following description.

[0061] As described in the following steps,

[0062] S210, determining a target data field in the file to be processed;

[0063] S220, converting the target data field to generate a byte array;

[0064] S230. Generate a first target partition based on the byte array and the total number of partitions of the current topic; and / or,

[0065] S240. Generate a second target partition based on the target data field, the byte array, and the total number of partitions of the current topic.

[0066] As an example, the optimization of the image topic for writing images to Kafka can use the image UID as the key of the message, so that the image will be written to different partitions of the image topic to achieve multi-threaded image processing; you can also enable a custom partitioner to determine which partition the image is written to; the above two methods can be used to solve the problem. The hospital can enable a corresponding strategy to solve the data hotspot problem caused by excessive data volume in a certain partition when writing images to Kafka, and can greatly improve the speed of consuming archived images.

[0067] As described in step S230, a first target partition is generated based on the byte array and the total number of partitions of the current topic.

[0068] In one embodiment of the present invention, the specific process of "generating the first target partition according to the byte array and the total number of partitions of the current topic" in step S230 can be further explained in combination with the following description.

[0069] As described in the following steps,

[0070] S310, generating a hash value according to the byte array and a preset algorithm, wherein the hash value is a positive number;

[0071] S320. Determine a first target partition based on the total number of partitions of the current topic and the hash value.

[0072] In one implementation, the image UID is used as the message key. This allows for multi-threaded image processing by writing the image to different partitions of the image topic. This is primarily due to the message key design, and the algorithm is the default for Kafka.

[0073] Use the KafkaTemplate.send method to write the image file to the image file topic. The algorithm formula for writing to the default partition of Kafka is as follows:

[0074] partition=Math.abs(Utils.murmur2(record.key().getBytes()))%

[0075] numPartitions;

[0076] Among them, partition: calculates the partition number to which the file will be written.

[0077] record.key().getBytes(): Convert the key of this message into a byte array (the image UID is the key of the message).

[0078] Utils.murmur2(...) : Generates hash values using the MurmurHash2 algorithm.

[0079] Math.abs(...) : Ensures that the hash value is a positive number.

[0080] %numPartitions: The total number of partitions of the current Image File Topic.

[0081] As described in step S240, a second target partition is generated based on the target data field, the byte array, and the total number of partitions of the current topic.

[0082] In one embodiment of the present invention, the specific process of "generating a second target partition according to the target data field, the byte array and the total number of partitions of the current topic" in step S240 can be further explained in combination with the following description.

[0083] As described in the following steps,

[0084] S410: Determine partition usage based on the total number of partitions of the current topic, the byte array, and the target data field;

[0085] S420: Determine a partition with the smallest data volume according to the partition usage, and mark the partition with the smallest data volume as a second target partition.

[0086] In a specific implementation, the partition usage of the current topic is queried, and the partition with the smallest data is found and written.

[0087] As described in step S120, the target writing mode is determined according to the file to be processed and the target partition.

[0088] In an embodiment of the present invention, the specific process of "determining the target writing mode according to the file to be processed and the target partition" in step S120 can be further explained in combination with the following description.

[0089] As described in the following steps,

[0090] S510, writing the to-be-processed file to the target partition, and determining whether there is degradation during the writing process;

[0091] S520: When the degradation situation exists, determine a target writing method according to the degradation situation.

[0092] In a specific implementation, it is used to implement fault tolerance, degradation, and recovery of image topics that write image files to Kafka.

[0093] As described in step S520, when the degradation situation exists, the target writing mode is determined according to the degradation situation.

[0094] It should be noted that the degradation situation includes the first degradation situation or the second degradation situation.

[0095] In an embodiment of the present invention, the specific process of "when the degradation situation exists, determining the target writing mode according to the degradation situation" in step S520 can be further explained in combination with the following description.

[0096] As described in the following steps,

[0097] S610: When the number of times the target image file is sent is greater than a first preset number, it is determined to be the first degradation situation, the target image file is stored in a first preset storage unit, and marked as the target writing mode of the first degradation situation; or

[0098] S620. When the number of abnormal times of obtaining the available topic list through the application program interface is greater than the preset second preset number, it is determined to be the second degradation situation, the target image file is stored in the second preset storage unit, and is marked as the target writing mode of the second degradation situation.

[0099] It should be noted that the high availability of the system is enhanced, thereby ensuring more stable data processing performance.

[0100] As an example, the first degradation situation: degradation triggered during writing: when an exception occurs when sending an image file to Kafka, and the same message cannot be successfully sent after 5 retries (that is, the first preset number of times, including but not limited to 5 times), the system should automatically switch to degradation mode, save the image file to the local disk, that is, the first preset storage unit, and no longer attempt to write to Kafka.

[0101] As an example, the second degradation situation: the heartbeat detection task for Kafka needs to be turned on when the service starts: if the topic list of KafkaConsumer.listTopics (all available topic metadata information in the Kafka cluster) obtained through the application interface is abnormal, and the number of consecutive times is greater than 5 times (that is, the second preset number, including but not limited to 5), it will automatically downgrade to disk writing mode.

[0102] In a specific implementation, in the second degradation case, if the topic list of KafkaConsumer.listTopics (all available topic metadata information in the Kafka cluster) is successfully obtained using the program interface, the mode will be restored to Kafka mode.

[0103] As described in step S140 , a target file is determined in the message queue according to a preset user requirement.

[0104] In an embodiment of the present invention, the specific process of "determining the target file in the message queue according to the preset user demand" in step S140 can be further explained in combination with the following description.

[0105] As described in the following steps,

[0106] S710, determining consumption information in the message queue according to preset user requirements;

[0107] S720: Generate parsed data from the consumption information using a preset parsing method;

[0108] S730: Generate target query structure information from the parsed data according to the index subject and the assembly format corresponding to the index subject;

[0109] S740: Determine a target file in the message queue according to the target query structure information.

[0110] In a specific implementation, when consuming images from a Kafka topic, the independent consumer thread only processes the archival storage of the images and does not assemble the index. It parses the information of the consumed image file (i.e., consumption information) through the dcm4che open source library (i.e., the preset parsing method), and then writes the parsed data (such as patient information, examination information, sequence information, and image-related information) to the Kafka index topic. The target query structure information is obtained by assembling the index through the consumption of the Kafka index topic, and then the target file is obtained in the message queue from the target query structure information.

[0111] Example 1

[0112] The above method can be used to visualize the following archive information:

[0113] 1) Archive status interface for consuming Kafka

[0114] To view the consumption of Kafka for archiving, you can view the consumption status of each partition.

[0115] 2) Consume Kafka archive logs

[0116] Abnormal data is displayed in red, and error information can be viewed.

[0117] 3) Failure list

[0118] For images that fail to be archived, you can view the reason for the failure and the images will be automatically resent. You can also manually resend them through the visual interface.

[0119] Application scenarios include the following:

[0120] Scenario 1: To meet the real-time archiving needs of PACS, imaging platforms, and cloud electronic film images for a single hospital or multiple hospitals, image files only need to be written to the Kafka queue according to this method.

[0121] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0122] Reference Figure 2 , shows a structural block diagram of a Kafka-based image rapid archiving system provided by an embodiment of the present application;

[0123] A Kafka-based rapid image archiving system, comprising:

[0124] A first determination module 210 is configured to determine a target partition based on the file to be processed and a preset theme;

[0125] A second determining module 220 is configured to determine a target writing mode based on the file to be processed and the target partition;

[0126] A first generating module 230 is configured to generate a message queue according to the target writing mode and the target partition;

[0127] A third determining module 240 is configured to determine a target file in the message queue according to a preset user requirement;

[0128] The second generating module 250 is configured to generate archiving information according to the target file, wherein the archiving information includes archiving status, archiving logs, and a failure list.

[0129] In one embodiment of the present invention, the first determining module 210 includes:

[0130] A first determining submodule, configured to determine a target data field in the file to be processed;

[0131] A first generating submodule, configured to convert the target data field to generate a byte array;

[0132] A second generating submodule is configured to generate a first target partition according to the byte array and the total number of partitions of the current topic; and / or,

[0133] The third generating submodule is used to generate a second target partition according to the target data field, the byte array and the total number of partitions of the current topic.

[0134] In one embodiment of the present invention, the second generating submodule includes:

[0135] A first generating unit, configured to generate a hash value according to the byte array and a preset algorithm, wherein the hash value is a positive number;

[0136] The first determining unit is used to determine the first target partition according to the total number of partitions of the current topic and the hash value.

[0137] In one embodiment of the present invention, the third generation submodule includes:

[0138] A second determining unit is configured to determine a partition usage according to the total number of partitions of the current topic, the byte array, and the target data field;

[0139] The third determining unit is configured to determine a partition with the smallest data volume according to the partition usage, and mark the partition with the smallest data volume as a second target partition.

[0140] In one embodiment of the present invention, the second determining module 220 includes:

[0141] A second determining submodule is configured to write the to-be-processed file into the target partition and determine whether there is degradation during the writing process;

[0142] The third determining submodule is configured to determine a target writing mode according to the degradation situation when the degradation situation exists.

[0143] In one embodiment of the present invention, the third determining submodule includes:

[0144] a first determining unit configured to determine that the target image file is in the first degradation situation when the number of times the target image file is sent is greater than a first preset number of times, store the target image file in a first preset storage unit, and mark it as a target write mode of the first degradation situation; or

[0145] The second determination unit is used to determine that it is the second degradation situation when the number of abnormal times of obtaining the available topic list through the application interface is greater than the preset second preset number, store the target image file in the second preset storage unit, and mark it as the target writing mode of the second degradation situation.

[0146] In one embodiment of the present invention, the third determining module 240 includes:

[0147] A fourth determining submodule, configured to determine consumption information in the message queue according to preset user requirements;

[0148] A fourth generating submodule, configured to generate parsed data from the consumption information using a preset parsing method;

[0149] a fifth generating submodule, configured to generate target query structure information from the parsed data according to index topics and an assembly format corresponding to the index topics;

[0150] The fifth determining submodule is configured to determine a target file in the message queue according to the target query structure information.

[0151] Reference Figure 3 , showing a computer device of a Kafka-based image rapid archiving method of the present invention, which may specifically include the following:

[0152] The computer device 12 is a general-purpose computing device. The components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0153] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor, or a local bus 18 that utilizes any of a variety of bus 18 architectures. Examples of such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.

[0154] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0155] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Figure 3 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of various embodiments of the present invention.

[0156] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in a memory. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0157] The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, a camera, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN)), a wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 3 Not shown, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34.

[0158] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a Kafka-based rapid image archiving method provided in an embodiment of the present invention.

[0159] That is, when the above-mentioned processing unit 16 executes the above-mentioned program, it realizes: determining the target partition based on the file to be processed and the preset subject; determining the target writing method based on the file to be processed and the target partition; generating a message queue based on the target writing method and the target partition; determining the target file in the message queue based on preset user requirements; generating archiving information based on the target file, wherein the archiving information includes archiving status, archiving log and failure list.

[0160] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a Kafka-based rapid image archiving method as provided in all embodiments of the present application:

[0161] That is, when the program is executed by the processor, it is implemented as follows: determining the target partition based on the file to be processed and the preset topic; determining the target writing method based on the file to be processed and the target partition; generating a message queue based on the target writing method and the target partition; determining the target file in the message queue based on preset user requirements; generating archiving information based on the target file, wherein the archiving information includes archiving status, archiving log and failure list.

[0162] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0163] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0164] The computer program code for performing the operations of the present invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0165] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0166] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0167] The above is a detailed introduction to the Kafka-based image rapid archiving method and system provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A Kafka-based image rapid archiving method, characterized in that: The method comprises: Determine the target partition based on the files to be processed and the preset topics; Determining a target writing mode according to the file to be processed and the target partition; generating a message queue according to the target writing mode and the target partition; Determining a target file in the message queue according to a preset user requirement; Archiving information is generated according to the target file, wherein the archiving information includes archiving status, archiving log and failure list.

2. The method according to claim 1, characterized in that The step of determining the target partition according to the file to be processed and the preset theme, wherein the target partition includes the first target partition and / or the second target partition, comprises: determining a target data field in the file to be processed; Converting the target data field to generate a byte array; Generate a first target partition according to the byte array and the total number of partitions of the current topic; and / or, A second target partition is generated according to the target data field, the byte array, and the total number of partitions of the current topic.

3. The method according to claim 2, characterized in that The step of generating a first target partition according to the byte array and the total number of partitions of the current topic includes: Generate a hash value based on the byte array and a preset algorithm, where the hash value is a positive number; The first target partition is determined according to the total number of partitions of the current topic and the hash value.

4. The method according to claim 2, characterized in that The step of generating a second target partition according to the target data field, the byte array, and the total number of partitions of the current topic comprises: Determine partition usage according to the total number of partitions of the current topic, the byte array, and the target data field; A partition with the smallest data volume is determined according to the partition usage, and the partition with the smallest data volume is marked as the second target partition.

5. The method according to claim 1, wherein The step of determining the target writing mode according to the file to be processed and the target partition includes: Writing the to-be-processed file to the target partition and determining whether there is degradation in the writing process; When the degradation situation exists, the target writing mode is determined according to the degradation situation.

6. The method according to claim 5, characterized in that When the degradation situation exists, determining a target write mode is generated according to the degradation situation, wherein the degradation situation includes a first degradation situation or a second degradation situation, including: When the number of times the target image file is sent is greater than a first preset number, it is determined to be the first degradation situation, the target image file is stored in a first preset storage unit, and marked as the target writing mode of the first degradation situation; or When the abnormal number of times the available topic list is obtained through the application interface is greater than the preset second preset number, it is determined to be the second degradation situation, the target image file is stored in the second preset storage unit, and marked as the target writing mode of the second degradation situation.

7. The method according to claim 1, characterized in that The step of determining the target file in the message queue according to the preset user requirements includes: Determining consumption information in the message queue according to preset user requirements; The consumption information is analyzed in a preset manner to generate analysis data; Generating target query structure information from the parsed data according to index topics and assembly formats corresponding to the index topics; A target file is determined in the message queue according to the target query structure information.

8. A Kafka-based image rapid archiving system, characterized in that: The system comprises: A first determination module is used to determine a target partition based on the file to be processed and a preset topic; A second determining module is used to determine a target writing mode according to the file to be processed and the target partition; A first generating module, configured to generate a message queue according to the target writing mode and the target partition; A third determining module is used to determine a target file in the message queue according to a preset user requirement; The second generating module is used to generate archiving information according to the target file, wherein the archiving information includes archiving status, archiving log and failure list.

9. A computer device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images