Big data server performance optimization method based on thread expansion

Abstract

The invention discloses a big data server performance optimization method based on thread expansion, and the method comprises the steps: continuously collecting road traffic video data through a multi-intersection camera, and uploading the video data to corresponding terminal equipment; and the terminal equipment analyzes videos uploaded by the multi-intersection camera into a frame sequence, selects multiple frames of optimal images from the frame sequence, and uploads the optimal images to the partition of the top corresponding to the big data platform kafka. According to the method, data consumption and processing are carried out in the mode that the single partition corresponds to the single thread and the main thread expands the multiple sub-threads, the concurrent processing capacity of the big data server for mass data is improved, the performance of the big data server is fully played, and the big data server can store and process mass concurrent data more stably and efficiently; according to the thread expansion, the main thread is exposed to the outside, and the sub-thread is hidden, so that the data service provided by the big data server is safer and more reliable, and is not easily attacked by the outside.

Description

Technical Field

[0001] This invention relates to the field of server performance optimization technology, specifically a method for optimizing the performance of big data servers based on thread expansion. Background Technology

[0002] With the rapid development of the Internet, a large amount of data has been generated, such as road traffic video data. Road traffic video data is an important information resource in the fields of urban traffic management, public safety assurance, and intelligent transportation system construction. It plays an important role in improving traffic management efficiency, ensuring traffic safety, and promoting the development of intelligent transportation systems.

[0003] With data storage and processing issues becoming increasingly prominent, how to process data efficiently has become a current research hotspot. As the core component of data processing, data servers play a crucial role in processing massive amounts of data. Currently, a large number of studies have optimized server performance by manually or automatically adjusting the operating parameters of data servers, making the server's processing of large amounts of data more stable and efficient.

[0004] Manually or automatically adjusting the operating parameters of a data server can improve server performance to some extent. However, the fixed performance of server hardware means that adjusting the operating parameters has very limited room for performance improvement and cannot fundamentally solve the problems caused by massive data concurrency. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a big data server performance optimization method based on thread expansion, which solves problems such as data latency, system saturation, and inability to fully utilize server performance when big data servers process massive amounts of data.

[0006] In view of the above problems, this application provides a big data server performance optimization method based on thread expansion. The method includes: continuously collecting road traffic video data from multiple intersection cameras and uploading the video data to the corresponding terminal device; the terminal device parses the video uploaded by the multiple intersection cameras into a frame sequence and selects the best multiple frames to upload to the partition of the corresponding topic of the big data platform Kafka.

[0007] The Java backend system uses a single-partition-main-thread consumption method to pull data from a single partition of a topic. The main thread extends multiple child threads and distributes the pulled data to these child threads for corresponding data business processing. After consuming the data, the child threads return the results to the main thread, which then returns the data consumption status to the corresponding topic partition based on the results. The topic partition then processes the information returned by the main thread accordingly.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] The big data server performance optimization method based on thread expansion improves the big data server's ability to handle massive amounts of concurrent data by using a single thread (main thread) for each partition and expanding the main thread to multiple child threads. This fully leverages the performance of the big data server, making the storage and processing of massive concurrent data more stable and efficient. Thread expansion also makes the data services provided by the big data server more secure and reliable by exposing the main thread and hiding the child threads, making it less susceptible to external attacks. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the performance optimization method for big data servers based on thread expansion according to the present invention.

[0011] Figure 2 This is a block diagram of the main thread extending child threads in this invention;

[0012] Figure 3 This is a flowchart of the partitioned thread data flow of the present invention. Detailed Implementation

[0013] To address the issues of data latency and system saturation in big data servers handling massive concurrent data, and the shortcomings of existing technologies, this invention provides a big data server performance optimization method based on thread expansion. This method aims to solve the problems of existing technologies failing to fully utilize server performance, resulting in data processing instability and even server performance failures when handling massive concurrent data.

[0014] To address the aforementioned technical problems, the overall approach of the technical solution provided in this application is as follows:

[0015] This application improves the concurrent processing capability of big data servers for massive amounts of data by using a single partition corresponding to a single thread (main thread) and extending the main thread to multiple child threads. This fully leverages the performance of big data servers and makes the storage and processing of massive concurrent data by big data servers more stable and efficient. Thread extension exposes the main thread to the outside world and hides the child threads, making the data services provided by big data servers more secure and reliable and less susceptible to external attacks.

[0016] After introducing the basic principles of this application, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should also be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all of them.

[0017] Example 1

[0018] like Figure 1-3 As shown, this application provides a big data server performance optimization method based on thread expansion. Taking road traffic video data as an example, the method includes:

[0019] S1. Multi-intersection cameras continuously collect road traffic video data and upload the video data to the corresponding terminal devices;

[0020] Multi-intersection cameras are devices used to monitor road traffic conditions. They are installed at multiple intersections to continuously collect video data. These cameras can capture the behavior and movement of vehicles, pedestrians, and other traffic participants, thereby providing data for traffic management and decision-making. This data has important application value in fields such as urban traffic management, public safety, vehicle navigation, and traffic planning.

[0021] S2. The terminal device parses the video uploaded by the multi-port camera into a frame sequence, and selects the best frames to upload to the corresponding topic partition of the big data platform Kafka.

[0022] To parse the video uploaded by multi-interface cameras into a frame sequence and select the best image to upload to the corresponding topic partition in Kafka on a big data platform, you can follow these steps:

[0023] 1. Use terminal equipment to parse the video uploaded by the multi-port camera and convert the video into a frame sequence. This can be achieved through a video decoder or image processing library.

[0024] 2. Select the optimal image from the frame sequence. The specific selection method can be adjusted according to actual needs, and the following factors can be considered:

[0025] *Image clarity: Selecting images with higher clarity allows for more accurate capture of road conditions.

[0026] *Object detection: If you need to detect objects on the road, you can select images where the objects are more obvious for subsequent processing.

[0027] *Ambient lighting: Selecting images with good lighting conditions can improve image readability and recognition accuracy.

[0028] *Special Events: If a special event (such as a traffic accident, vehicle violation, etc.) occurs, you can select the relevant frames to upload for subsequent analysis.

[0029] 3. Upload the selected best images to the corresponding topic partition of the Kafka platform. This requires using a Kafka client library to send the image data as a message to the specified topic and partition.

[0030] When a terminal device sends message data, it performs a modulo operation on the ID corresponding to the message data based on the total number of partitions in the topic. IDs with the same modulo result will have their message data sent to the same partition. Messages are ordered within each partition, and one partition corresponds to one consumer, ensuring the order of message consumption.

[0031] Message data is assigned to a specific partition, and the total number of partitions can be adjusted according to actual needs. Messages with the same ID are always sent to the same partition, which ensures that messages within the same partition are ordered.

[0032] S3, the Java backend system pulls data from a single partition of a topic using a single partition-single thread (main thread) consumption method;

[0033] S4. The main thread expands into multiple sub-threads and distributes the fetched data to these sub-threads for corresponding data business processing.

[0034] The main thread expands into multiple child threads as follows: The main thread performs a hash operation on the retrieved message ID, and then performs a modulo operation on the result based on the specified total number of child threads. The modulo-ordered message is then sent to the corresponding message queue for the child threads to consume.

[0035] S5. After consuming data, the child thread returns the result information to the main thread. The main thread then returns the data consumption status to the corresponding topic partition based on the result returned by the child thread.

[0036] After a child thread successfully consumes message data from the main thread, it returns a success message to the main thread. If an error occurs during data consumption by the child thread, it returns the topic corresponding to the message data to the main thread.

[0037] The main thread processes the information returned by the child thread accordingly. If it receives a message indicating successful consumption, it returns the message to the corresponding topic partition. If it receives a message indicating an error in consumption, the main thread will overwrite the data in the corresponding topic partition.

[0038] S6, topic partitions process the information returned by the main thread accordingly.

[0039] The topic partition processes the information returned by the main thread as follows: after receiving a message indicating successful consumption by the main thread, the topic partition deletes the corresponding topic data; if the topic partition receives a topic returned by the main thread, the corresponding topic data is retained.

[0040] Although embodiments of the present invention have been shown and described, these specific embodiments are merely explanations of the invention and are not intended to limit it. The specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. After reading this specification, those skilled in the art may make modifications, substitutions, and variations to the embodiments as needed without departing from the principles and spirit of the invention, but such modifications, substitutions, and variations are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A method for optimizing the performance of a big data server based on thread expansion, characterized in that: The method includes the following steps: S1. Multi-intersection cameras continuously collect road traffic video data and upload the video data to the corresponding terminal devices; S2. The terminal device parses the video uploaded by the multi-port camera into a frame sequence, and selects the best frames to upload to the corresponding topic partition of the big data platform Kafka. S3, the Java backend system pulls data from a single partition of a topic using a single partition-main thread consumption method; S4. The main thread expands into multiple sub-threads and distributes the fetched data to these sub-threads for corresponding data business processing. S5. After consuming data, the child thread returns the result information to the main thread. The main thread then returns the data consumption status to the corresponding topic partition based on the result returned by the child thread. S6, topic partitions process the information returned by the main thread accordingly.

2. The method for optimizing the performance of a big data server based on thread expansion according to claim 1, characterized in that: In step S2, when the terminal device sends message data, it performs a modulo operation on the ID corresponding to the message data based on the total number of partitions of the topic. IDs with the same modulo result will send the message data to the same partition. The messages are ordered within the partition, and one partition corresponds to one consumer.

3. The method for optimizing the performance of a big data server based on thread expansion according to claim 1, characterized in that: In step S4, the main thread expands into multiple child threads as follows: the main thread performs a hash operation on the retrieved message ID, performs a modulo operation on the result based on the specified total number of child threads, and sends the modulo-ordered message to the corresponding message queue for the child threads to consume.

4. The method for optimizing the performance of a big data server based on thread expansion according to claim 1, characterized in that: In step S5, after the sub-thread successfully consumes message data from the main thread, it returns the successful consumption information to the main thread. If the sub-thread encounters an error in consuming the data, it returns the topic corresponding to the message data to the main thread. The main thread processes the information returned by the child thread accordingly. If it receives a message indicating successful consumption, it returns the message to the corresponding topic partition. If it receives a message indicating an error in consumption, the main thread will overwrite the data in the corresponding topic partition.

5. The method for optimizing the performance of a big data server based on thread expansion according to claim 1, characterized in that: In step S6, the topic partition processes the information returned by the main thread as follows: after receiving the information that the main thread has successfully consumed the information, the topic partition deletes the corresponding topic data; if the topic partition receives the topic returned by the main thread, the corresponding topic data is retained.

Images