A kafka-based method and system for rapid image archiving

Abstract

The application provides a Kafka-based image rapid archiving method and system, the method comprising: determining a target partition according to a to-be-processed file and a preset theme; determining a target writing mode according to the to-be-processed file and the target partition; generating a message queue according to the target writing mode and the target partition; determining a target file in the message queue according to a preset user demand; and generating archiving information according to the target file, wherein the archiving information comprises archiving conditions, archiving logs and a failure list. The archiving speed is faster, the fault tolerance mechanism is more perfect, and the operation and maintenance and problem troubleshooting process are greatly simplified.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method and system for rapid image archiving based on Kafka. Background Technology

[0002] In the past year or two, the pace of cloud computing deployment in the medical industry has accelerated significantly. At the same time, IT giants are partnering with medical system integrators to develop and design medical clouds as an important goal, and are preparing to build medical imaging cloud platforms in various cities and regions to achieve data interconnection. However, the rapid archiving of image files has become a problem when medical data is uploaded to the cloud.

[0003] Existing fast archiving technologies involve writing images to Kafka, which can lead to hotspots in the partition queues and queue congestion, resulting in untimely archiving. Furthermore, Kafka server disks are prone to filling up, and if the Kafka cluster goes down, the entire service becomes unusable, with no fault tolerance mechanism. In addition, there is no visual interface, which increases the difficulty of operation and maintenance, makes it difficult to troubleshoot problems, and prevents technical personnel from troubleshooting. Summary of the Invention

[0004] In view of the aforementioned problems, this application is proposed to provide a Kafka-based method and system for rapid image archiving that overcomes or at least partially solves the aforementioned problems, comprising:

[0005] A method for fast image archiving based on Kafka, the method comprising:

[0006] Determine the target partition based on the file to be processed and the preset theme;

[0007] The target write method is determined based on the file to be processed and the target partition;

[0008] A message queue is generated based on the target write method and the target partition;

[0009] The target file is determined in the message queue according to the preset user requirements;

[0010] Archive information is generated based on the target file, wherein the archive information includes archive status, archive logs, and a failure list.

[0011] Further, the step of determining the target partition based on the file to be processed and the preset topic, wherein the target partition includes a first target partition and / or a second target partition, includes:

[0012] Determine the target data fields in the file to be processed;

[0013] The target data field is converted to generate a byte array;

[0014] Generate a first target partition based on the byte array and the total number of partitions in the current topic; and / or,

[0015] A second target partition is generated based on the target data field, the byte array, and the total number of partitions in the current topic.

[0016] Further, the step of generating the first target partition based on the byte array and the total number of partitions in the current topic includes:

[0017] A hash value is generated based on the byte array and a preset algorithm, and the hash value is a positive number;

[0018] The first target partition is determined based on the total number of partitions in the current topic and the hash value.

[0019] Further, the step of generating a second target partition based on the target data field, the byte array, and the total number of partitions in the current topic includes:

[0020] The partition usage is determined based on the total number of partitions for the current topic, the byte array, and the target data field;

[0021] Based on the partition usage, determine the partition with the smallest data volume and mark the partition with the smallest data volume as the second target partition.

[0022] Furthermore, the step of determining the target write method based on the file to be processed and the target partition includes:

[0023] Write the file to be processed to the target partition and determine whether there is any degradation during the writing process;

[0024] When the aforementioned degradation situation exists, the target write method is determined based on the degradation situation.

[0025] Furthermore, when the aforementioned degradation situation exists, a target write method is generated based on the degradation situation, wherein the degradation situation includes either a first degradation situation or a second degradation situation, comprising the following steps:

[0026] When the number of times the target image file is sent exceeds a first preset number, it is determined to be the first degradation case. The target image file is then stored in a first preset storage unit and marked as the target write method for the first degradation case; or,

[0027] When the number of times the available topic list is obtained through the application programming interface exceeds a preset second number, it is determined to be the second degradation case. The target image file is then stored in the second preset storage unit and marked as the target write method for the second degradation case.

[0028] Furthermore, the step of determining the target file in the message queue based on preset user requirements includes:

[0029] Consumption information is determined in the message queue based on preset user requirements;

[0030] The consumption information is used to generate parsed data through a preset parsing method;

[0031] The parsed data is used to generate target query structure information according to the index topic and the assembly format corresponding to the index topic;

[0032] The target file is determined in the message queue based on the target query structure information.

[0033] An embodiment of this application also discloses a Kafka-based rapid image archiving system, the system comprising:

[0034] The first determination module is used to determine the target partition based on the file to be processed and the preset theme;

[0035] The second determining module is used to determine the target writing method based on the file to be processed and the target partition;

[0036] The first generation module is used to generate a message queue based on the target writing method and the target partition;

[0037] The third determination module is used to determine the target file in the message queue according to preset user requirements;

[0038] The second generation module is used to generate archive information based on the target file, wherein the archive information includes archive status, archive logs, and a failure list.

[0039] An embodiment of this application also discloses a computer device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the steps of a Kafka-based rapid image archiving method as described above.

[0040] An embodiment of this application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a Kafka-based rapid image archiving method as described above.

[0041] This application has the following advantages:

[0042] In the embodiments of this application, compared with the existing technology where "existing fast archiving technology for images writing to Kafka has hot spots in the partition queue, but there is a possibility of queue congestion, resulting in untimely archiving, and the Kafka server disk is prone to fullness. Once the Kafka cluster goes down, the entire service becomes unusable, and there is no fault tolerance mechanism; in addition, there is no visual interface, which increases the difficulty of operation and maintenance, makes it difficult to troubleshoot problems, and makes it impossible for technicians to troubleshoot", this application provides a solution for "fast archiving of image files based on Kafka", specifically: "a method for fast image archiving based on Kafka, the method comprising: determining a target partition based on the file to be processed and a preset topic; determining a target writing method based on the file to be processed and the target partition; generating a message queue based on the target writing method and the target partition; determining a target file in the message queue according to preset user requirements; generating archiving information based on the target file, wherein the archiving information includes archiving status, archiving logs, and a failure list." The method addresses the following issues by "determining the target partition based on the file to be processed and the preset topic; determining the target write method based on the file to be processed and the target partition; generating a message queue based on the target write method and the target partition; determining the target file in the message queue based on preset user requirements; and generating archive information based on the target file, wherein the archive information includes archive status, archive logs, and a failure list." This solves the problems of "existing fast archiving technologies where image writing to Kafka suffers from partition queue hotspots and queue congestion, leading to untimely archiving; Kafka server disks are prone to filling up, and if the Kafka cluster crashes, the entire service becomes unusable, with no fault tolerance mechanism; furthermore, the lack of a visual interface increases the difficulty of operation and maintenance, makes troubleshooting difficult, and prevents technical personnel from troubleshooting," thus achieving "a solution for image data writing to Kafka and..." The process of consuming Kafka has been optimized, with enhanced fault tolerance mechanisms added to ensure the stability and reliability of data processing. A user-friendly management interface is provided to facilitate monitoring and management of the entire process. Regarding image writing to Kafka, the partitioning strategy and algorithm have been optimized to effectively avoid queue congestion (hotspot partitioning) and significantly improve the speed and efficiency of archiving. The risk of Kafka disk space running out quickly has also been addressed, improving system high availability and ensuring more stable data processing performance. A new index topic has been added to specifically handle indexes, while image topics are solely responsible for image processing. Visualization has been implemented when archiving Kafka images, allowing users to view the current consumption status and the interface for handling consumption failures and retrying, greatly simplifying the operation and troubleshooting process. Attached Figure Description

[0043] To more clearly illustrate the technical solution of this application, the drawings used in the description of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the steps of a Kafka-based rapid image archiving method according to an embodiment of this application;

[0045] Figure 2 This is a structural block diagram of a Kafka-based rapid image archiving system provided in one embodiment of this application;

[0046] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention;

[0047] Figure 4 This is a flowchart illustrating one embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, features, and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0049] Reference Figure 1 and Figure 4 The diagram shows a flowchart and schematic diagram of a method for rapid image archiving based on Kafka, provided in an embodiment of this application.

[0050] A method for fast image archiving based on Kafka, the method comprising:

[0051] S110. Determine the target partition based on the file to be processed and the preset theme;

[0052] S120. Determine the target writing method based on the file to be processed and the target partition;

[0053] S130. Generate a message queue according to the target writing method and the target partition;

[0054] S140. Determine the target file in the message queue according to the preset user requirements;

[0055] S150. Generate archive information based on the target file, wherein the archive information includes archive status, archive log and failure list.

[0056] In the embodiments of this application, compared with the existing technology where "existing fast archiving technology for images writing to Kafka has hot spots in the partition queue, but there is a possibility of queue congestion, resulting in untimely archiving, and the Kafka server disk is prone to fullness. Once the Kafka cluster goes down, the entire service becomes unusable, and there is no fault tolerance mechanism; in addition, there is no visual interface, which increases the difficulty of operation and maintenance, makes it difficult to troubleshoot problems, and makes it impossible for technicians to troubleshoot", this application provides a solution for "fast archiving of image files based on Kafka", specifically: "a method for fast image archiving based on Kafka, the method comprising: determining a target partition based on the file to be processed and a preset topic; determining a target writing method based on the file to be processed and the target partition; generating a message queue based on the target writing method and the target partition; determining a target file in the message queue according to preset user requirements; generating archiving information based on the target file, wherein the archiving information includes archiving status, archiving logs, and a failure list." The method addresses the following issues by "determining the target partition based on the file to be processed and the preset topic; determining the target write method based on the file to be processed and the target partition; generating a message queue based on the target write method and the target partition; determining the target file in the message queue based on preset user requirements; and generating archive information based on the target file, wherein the archive information includes archive status, archive logs, and a failure list." This solves the problems of "existing fast archiving technologies where image writing to Kafka suffers from partition queue hotspots and queue congestion, leading to untimely archiving; Kafka server disks are prone to filling up, and if the Kafka cluster crashes, the entire service becomes unusable, with no fault tolerance mechanism; furthermore, the lack of a visual interface increases the difficulty of operation and maintenance, makes troubleshooting difficult, and prevents technical personnel from troubleshooting," thus achieving "a solution for image data writing to Kafka and..." The process of consuming Kafka has been optimized, with enhanced fault tolerance mechanisms added to ensure the stability and reliability of data processing. A user-friendly management interface is provided to facilitate monitoring and management of the entire process. Regarding image writing to Kafka, the partitioning strategy and algorithm have been optimized to effectively avoid queue congestion (hotspot partitioning) and significantly improve the speed and efficiency of archiving. The risk of Kafka disk space running out quickly has also been addressed, improving system high availability and ensuring more stable data processing performance. A new index topic has been added to specifically handle indexes, while image topics are solely responsible for image processing. Visualization has been implemented when archiving Kafka images, allowing users to view the current consumption status and the interface for handling consumption failures and retrying, greatly simplifying the operation and troubleshooting process.

[0057] The following will further describe a Kafka-based method for rapid image archiving in this exemplary embodiment.

[0058] As described in step S110, the target partition is determined based on the file to be processed and the preset theme.

[0059] It should be noted that the target partition includes a first target partition and / or a second target partition.

[0060] In one embodiment of the present invention, the specific process of "determining the target partition based on the file to be processed and the preset theme" in step S110 can be further explained in conjunction with the following description.

[0061] As described in the following steps

[0062] S210. Determine the target data field in the file to be processed;

[0063] S220. Convert the target data field to generate a byte array;

[0064] S230. Generate a first target partition based on the byte array and the total number of partitions in the current topic; and / or,

[0065] S240. Generate a second target partition based on the target data field, the byte array, and the total number of partitions in the current topic.

[0066] As an example, optimizing image writing to Kafka image topics involves using the image UID as the message key. This allows images to be written to different partitions within the image topic, enabling multi-threaded image processing. Alternatively, a custom partitioner can be enabled to specify which partition an image is written to. Both methods can be used to address the data hotspot problem caused by excessive data volume in a particular partition when writing images to Kafka, and can also significantly improve the speed of consuming archived images.

[0067] As described in step S230, a first target partition is generated based on the byte array and the total number of partitions in the current topic.

[0068] In one embodiment of the present invention, the specific process of "generating a first target partition based on the byte array and the total number of partitions of the current topic" in step S230 can be further explained in conjunction with the following description.

[0069] As described in the following steps

[0070] S310. Generate a hash value based on the byte array and a preset algorithm, wherein the hash value is a positive number;

[0071] S320. Determine the first target partition based on the total number of partitions in the current topic and the hash value.

[0072] In one specific implementation, the image UID is used as the message key. This allows the image to be written to different partitions of the image topic, enabling multi-threaded image processing. The key is the design of the message key; the algorithm used is Kafka's default.

[0073] The image file is written to the image file topic using the KafkaTemplate.send method. The algorithm formula for Kafka's default partition write is as follows:

[0074] partition=Math.abs(Utils.murmur2(record.key().getBytes()))%

[0075] numPartitions;

[0076] Among them, partition: calculates which partition number the file will be written to.

[0077] record.key().getBytes(): Converts the key of this message into a byte array (the image UID is the key of the message).

[0078] Utils.murmur2(...): Generates a hash value using the MurmurHash2 algorithm.

[0079] Math.abs(...): Ensures that the hash value is positive.

[0080] %numPartitions: The total number of partitions in the current Image File Theme.

[0081] As described in step S240, a second target partition is generated based on the target data field, the byte array, and the total number of partitions in the current topic.

[0082] In one embodiment of the present invention, the specific process of "generating a second target partition based on the target data field, the byte array and the total number of partitions of the current topic" in step S240 can be further explained in conjunction with the following description.

[0083] As described in the following steps

[0084] S410. Determine the partition usage based on the total number of partitions for the current topic, the byte array, and the target data field;

[0085] S420. Based on the partition usage, determine the partition with the smallest data volume and mark the partition with the smallest data volume as the second target partition.

[0086] In one specific implementation, the partition usage of the current topic is queried, the partition with the smallest amount of data is found, and then the data is written to.

[0087] As described in step S120, the target writing method is determined based on the file to be processed and the target partition.

[0088] In one embodiment of the present invention, the specific process of "determining the target writing method based on the file to be processed and the target partition" in step S120 can be further explained in conjunction with the following description.

[0089] As described in the following steps

[0090] S510. Write the file to be processed to the target partition and determine whether there is any degradation during the writing process;

[0091] S520. When the aforementioned degradation situation exists, the target write method is determined based on the degradation situation.

[0092] In one specific implementation, it is used to implement image topic fault tolerance, degradation, and recovery for writing image files to Kafka.

[0093] As described in step S520, when the degradation situation exists, the target writing method is determined based on the degradation situation.

[0094] It should be noted that the downgrade scenarios include either the first downgrade scenario or the second downgrade scenario.

[0095] In one embodiment of the present invention, the specific process of "determining the target writing method based on the degradation situation when the degradation situation exists" in step S520 can be further explained in conjunction with the following description.

[0096] As described in the following steps

[0097] S610. When the number of times the target image file is sent exceeds a first preset number, it is determined to be the first degradation situation. The target image file is stored in the first preset storage unit and marked as the target write mode of the first degradation situation; or,

[0098] S620. When the number of abnormal occurrences of obtaining the list of available topics through the application programming interface exceeds a preset second preset number, it is determined to be the second degradation situation. The target image file is stored in the second preset storage unit and marked as the target writing mode of the second degradation situation.

[0099] It should be noted that the system's high availability has been enhanced, thereby ensuring more stable data processing performance.

[0100] As an example, the first degradation case is degradation triggered during writing: when an exception is encountered when sending an image file to Kafka, and the same message still cannot be sent successfully after 5 retries (i.e., the first preset number, including but not limited to 5), the system should automatically switch to degradation mode, save the image file to the local disk, i.e., the first preset storage unit, and no longer attempt to write to Kafka.

[0101] As an example, the second degradation scenario is as follows: When the service starts, a heartbeat detection task for Kafka needs to be enabled: If the list of topics obtained through the application programming interface, KafkaConsumer.listTopics (database information of all available topics in the Kafka cluster), is abnormal and the number of consecutive occurrences is greater than 5 (that is, the second preset number, including but not limited to 5), it will automatically degrade to disk write mode.

[0102] In one specific implementation, in the second downgrade scenario, if the program interface successfully retrieves the list of topics KafkaConsumer.listTopics (database information of all available topics in the Kafka cluster), the recovery mode will be changed to Kafka mode.

[0103] As described in step S140, the target file is determined in the message queue according to the preset user requirements.

[0104] In one embodiment of the present invention, the specific process of "determining the target file in the message queue according to preset user requirements" in step S140 can be further explained in conjunction with the following description.

[0105] As described in the following steps

[0106] S710. Determine consumption information in the message queue according to preset user requirements;

[0107] S720. Generate parsed data from the consumption information using a preset parsing method;

[0108] S730. Generate target query structure information from the parsed data according to the index topic and the assembly format corresponding to the index topic;

[0109] S740. Determine the target file in the message queue based on the target query structure information.

[0110] In one specific implementation, when consuming images from a Kafka topic, the independent consumption thread only handles the archiving and storage of images and does not assemble the index. It parses the information (i.e., consumption information) of the consumed image files using the dcm4che open-source library (i.e., the default parsing method), and then writes the parsed data (such as patient information, examination information, sequence information, and image-related information) to the Kafka index topic. By consuming the Kafka index topic, the target query structure information is obtained through index assembly, and then the target file is obtained from the message queue based on the target query structure information.

[0111] Example 1

[0112] The above method can be used to visualize the following archived information:

[0113] 1) Interface for consuming Kafka archive status

[0114] To view the consumption of Kafka archives, you can see the consumption status for each partition.

[0115] 2) Consume Kafka archived logs

[0116] Abnormal data is indicated in red, and error messages can be viewed.

[0117] 3) Failure list

[0118] For images that fail to be archived for consumption, you can view the reason for the failure and it will be automatically resent. You can also manually resend them through a visual interface.

[0119] The application scenarios include the following:

[0120] Scenario 1: To meet the real-time archiving needs of PACS, imaging platforms, and cloud electronic film images for single or multiple hospitals, the image files only need to be written to the Kafka queue using this method.

[0121] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0122] Reference Figure 2 The diagram shows a structural block diagram of a Kafka-based rapid image archiving system provided in an embodiment of this application.

[0123] A Kafka-based image rapid archiving system, the system comprising:

[0124] The first determining module 210 is used to determine the target partition based on the file to be processed and the preset theme;

[0125] The second determining module 220 is used to determine the target writing method based on the file to be processed and the target partition;

[0126] The first generation module 230 is used to generate a message queue based on the target writing method and the target partition;

[0127] The third determining module 240 is used to determine the target file in the message queue according to preset user requirements;

[0128] The second generation module 250 is used to generate archive information based on the target file, wherein the archive information includes archive status, archive logs and failure list.

[0129] In one embodiment of the present invention, the first determining module 210 includes:

[0130] The first determining submodule is used to determine the target data field in the file to be processed;

[0131] The first generation submodule is used to convert the target data field to generate a byte array;

[0132] The second generation submodule is used to generate a first target partition based on the byte array and the total number of partitions in the current topic; and / or,

[0133] The third generation submodule is used to generate a second target partition based on the target data field, the byte array, and the total number of partitions in the current topic.

[0134] In one embodiment of the present invention, the second generation submodule includes:

[0135] The first generation unit is used to generate a hash value based on the byte array and a preset algorithm, wherein the hash value is a positive number;

[0136] The first determining unit is used to determine the first target partition based on the total number of partitions of the current topic and the hash value.

[0137] In one embodiment of the present invention, the third generation submodule includes:

[0138] The second determining unit is used to determine the partition usage based on the total number of partitions of the current topic, the byte array, and the target data field;

[0139] The third determining unit is used to determine the partition with the smallest data volume based on the partition usage, and mark the partition with the smallest data volume as the second target partition.

[0140] In one embodiment of the present invention, the second determining module 220 includes:

[0141] The second determining submodule is used to write the file to be processed to the target partition and determine whether there is a degradation situation during the writing process;

[0142] The third determining submodule is used to determine the target writing method based on the degradation situation when the degradation situation exists.

[0143] In one embodiment of the present invention, the third determining submodule includes:

[0144] The first determination unit is configured to, when the number of times the target image file is sent exceeds a first preset number, determine that it is the first degradation situation, store the target image file in a first preset storage unit, and mark it as the target write mode of the first degradation situation; or,

[0145] The second determination unit is used to determine the second degradation situation when the number of abnormal times of obtaining the list of available topics through the application programming interface is greater than a preset second preset number, and to store the target image file in the second preset storage unit and mark it as the target writing mode of the second degradation situation.

[0146] In one embodiment of the present invention, the third determining module 240 includes:

[0147] The fourth determination submodule is used to determine consumption information in the message queue based on preset user requirements;

[0148] The fourth generation submodule is used to generate parsed data from the consumption information using a preset parsing method;

[0149] The fifth generation submodule is used to generate target query structure information from the parsed data according to the index topic and the assembly format corresponding to the index topic;

[0150] The fifth determination submodule is used to determine the target file in the message queue based on the target query structure information.

[0151] Reference Figure 3 The computer device illustrating a Kafka-based method for rapid image archiving according to the present invention may specifically include the following:

[0152] The computer device 12 described above is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0153] Bus 18 refers to one or more of several types of bus 18 architectures, including memory bus 18 or memory controller, peripheral bus 18, graphics acceleration port, processor, or local bus 18 using any of the various bus 18 architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus 18, Micro Channel Architecture (MAC) bus 18, Enhanced ISA bus 18, Audio / Video Electronics Standards Association (VESA) local bus 18, and Peripheral Component Interconnect (PCI) bus 18.

[0154] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0155] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard disk drive"). Figure 3 Not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of the embodiments of the present invention.

[0156] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory. Such program modules 42 include—but are not limited to—an operating system, one or more application programs, other program modules 42, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0157] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, camera, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN)), wide area network (WAN), and / or public networks (e.g., the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 3 Not shown, it can be combined with computer device 12 to use other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing unit 16, external disk drive array, RAID system, tape drive and data backup storage system 34, etc.

[0158] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a Kafka-based rapid image archiving method provided in this embodiment of the invention.

[0159] That is, when the processing unit 16 executes the above program, it performs the following: determining the target partition based on the file to be processed and the preset topic; determining the target writing method based on the file to be processed and the target partition; generating a message queue based on the target writing method and the target partition; determining the target file in the message queue based on the preset user requirements; and generating archive information based on the target file, wherein the archive information includes archive status, archive logs, and a failure list.

[0160] In this embodiment of the invention, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a Kafka-based method for rapid image archiving as provided in all embodiments of this application:

[0161] That is, when the program is executed by the processor, it performs the following: determining the target partition based on the file to be processed and the preset topic; determining the target write method based on the file to be processed and the target partition; generating a message queue based on the target write method and the target partition; determining the target file in the message queue based on preset user requirements; and generating archive information based on the target file, wherein the archive information includes archive status, archive logs, and a failure list.

[0162] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-to-signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0163] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0164] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider). The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0165] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0166] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0167] The above provides a detailed description of the Kafka-based rapid image archiving method and system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for rapid image archiving based on Kafka, characterized in that, The method includes: Determine the target partition based on the file to be processed and the preset theme; The target write method is determined based on the file to be processed and the target partition; A message queue is generated based on the target write method and the target partition; The target file is determined in the message queue according to the preset user requirements; The process involves: generating archive information based on the target file, including archive status, archive logs, and a failure list; determining target data fields in the file to be processed; converting the target data fields to generate a byte array; generating a first target partition based on the byte array and the total number of partitions in the current topic; and / or generating a second target partition based on the target data fields, the byte array, and the total number of partitions in the current topic. The step of generating the first target partition based on the byte array and the total number of partitions in the current topic includes: generating a hash value based on the byte array and a preset algorithm, where the hash value is positive; determining the first target partition based on the total number of partitions in the current topic and the hash value. The step of generating the second target partition based on the target data fields, the byte array, and the total number of partitions in the current topic includes: determining partition usage based on the total number of partitions in the current topic, the byte array, and the target data fields; determining the partition with the smallest data volume based on the partition usage, and marking the partition with the smallest data volume as the second target partition.

2. The method according to claim 1, characterized in that, The step of determining the target write method based on the file to be processed and the target partition includes: Write the file to be processed to the target partition and determine whether there is any degradation during the writing process; When the aforementioned degradation situation exists, the target write method is determined based on the degradation situation.

3. The method according to claim 2, characterized in that, When the aforementioned degradation situation exists, a target write method is generated based on the degradation situation, wherein the degradation situation includes either a first degradation situation or a second degradation situation, and the steps include: When the number of times the target image file is sent exceeds a first preset number, it is determined to be the first degradation case. The target image file is then stored in a first preset storage unit and marked as the target write method for the first degradation case; or, When the number of times the available topic list is obtained through the application programming interface exceeds a preset second number, it is determined to be the second degradation case. The target image file is then stored in the second preset storage unit and marked as the target write method for the second degradation case.

4. The method according to claim 1, characterized in that, The step of determining the target file in the message queue according to preset user requirements includes: Consumption information is determined in the message queue based on preset user requirements; The consumption information is used to generate parsed data through a preset parsing method; The parsed data is used to generate target query structure information according to the index topic and the assembly format corresponding to the index topic; The target file is determined in the message queue based on the target query structure information.

5. A Kafka-based rapid image archiving system, characterized in that, The system includes: The first determination module is used to determine the target partition based on the file to be processed and the preset theme; The second determining module is used to determine the target writing method based on the file to be processed and the target partition; The first generation module is used to generate a message queue based on the target writing method and the target partition; The third determination module is used to determine the target file in the message queue according to preset user requirements; The second generation module is used to generate archive information based on the target file, wherein the archive information includes archive status, archive logs, and a failure list; determine target data fields in the file to be processed; convert the target data fields to generate a byte array; generate a first target partition based on the byte array and the total number of partitions in the current topic; and / or generate a second target partition based on the target data fields, the byte array, and the total number of partitions in the current topic; the step of generating the first target partition based on the byte array and the total number of partitions in the current topic includes: generating a hash value based on the byte array and a preset algorithm, wherein the hash value is a positive number; determining the first target partition based on the total number of partitions in the current topic and the hash value; the step of generating the second target partition based on the target data fields, the byte array, and the total number of partitions in the current topic includes: determining the partition usage based on the total number of partitions in the current topic, the byte array, and the target data fields; determining the partition with the smallest data volume based on the partition usage, and marking the partition with the smallest data volume as the second target partition.

6. A computer device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images