Data processing method and electronic device

Abstract

The application discloses a data processing method and an electronic device, and relates to the technical field of computers, and comprises the following steps: alternately receiving first transmission data by using a first target buffer and a second target buffer, wherein the first transmission data is data read from a source buffer; storing second transmission data in a circular buffer, wherein the second transmission data is data obtained by preprocessing the first transmission data; determining a scheduling mode of the second transmission data according to a data feature of the first transmission data, wherein the scheduling mode is one of a fixed scheduling mode and a dynamic scheduling mode; and reading and processing the second transmission data from the circular buffer according to the scheduling mode. Through the key steps of alternately receiving data by using the target buffers, storing the data in the circular buffer after preprocessing, and flexibly selecting the scheduling mode according to a business scenario, efficient parallelism of data transmission and processing is realized, and resource consumption of a system bus is reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more particularly to a data processing method and an electronic device. Background Technology

[0002] Existing technologies may be inefficient in the automatic management and switching of buffers, especially when data traffic fluctuates significantly. Double-buffering or circular-buffering mechanisms are prone to untimely buffer switching, leading to data processing delays or data overflows. In systems with multiple devices sharing a bus, existing technologies may not be able to effectively and dynamically adjust the allocation of bus resources, resulting in uneven resource distribution and low transmission efficiency for some devices. Furthermore, existing technologies for implementing DMA double-buffering and circular-buffering may not be fully compatible with all types of devices, especially those with special data transmission requirements, which may require additional configuration and optimization.

[0003] Therefore, with the continuous increase in data volume and the improvement in transmission speed requirements, the traditional DMA transmission method still occupies the system bus and remains a bottleneck restricting system performance. Summary of the Invention

[0004] This application provides a data processing method and an electronic device to at least solve the problem in the related art that, in high data transfer rate environments, traditional DMA transfer methods occupy bus resources for a long time, especially in single-buffered mode, where bus resources are monopolized by a single DMA operation for a long time, severely limiting other devices' access to the bus and reducing the overall utilization of the bus.

[0005] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: alternately receiving first transmission data using a first target buffer and a second target buffer, wherein the first transmission data is data read from a source buffer; transferring second transmission data to a circular buffer, wherein the second transmission data is data obtained by preprocessing the first transmission data; determining a scheduling mode for the second transmission data based on the data characteristics of the first transmission data, wherein the scheduling mode is one of the following: a fixed scheduling mode and a dynamic scheduling mode; and reading and processing the second transmission data from the circular buffer according to the scheduling mode.

[0006] According to another aspect of the embodiments of this application, an electronic device is also provided, including: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing the data processing methods described in any one of the foregoing.

[0007] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data processing methods.

[0008] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the steps of the data processing method described in any one of the above-described embodiments.

[0009] This application utilizes a first target buffer and a second target buffer to alternately receive first transmission data, wherein the first transmission data is data read from the source buffer; second transmission data is transferred to a circular buffer, wherein the second transmission data is data obtained by preprocessing the first transmission data; a scheduling method for the second transmission data is determined, wherein the scheduling method is one of the following: fixed scheduling method and dynamic scheduling method; the second transmission data is read from and processed from the circular buffer according to the scheduling method. Through these key steps of alternately using the target buffer to receive data, preprocessing and transferring the data to the circular buffer, and flexibly selecting the scheduling method according to the business scenario, efficient parallel data transmission and processing are achieved, reducing the resource consumption of the system bus, improving the real-time performance and stability of data processing, and solving the problem in related technologies where, in high data transmission rate environments, traditional DMA transmission methods occupy bus resources for a long time, especially in single-buffer mode, where bus resources are monopolized by a single DMA operation for a long time, severely limiting other devices' access to the bus and reducing the overall utilization of the bus. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of a single buffer based on existing technology;

[0012] Figure 2 This is a schematic diagram of double buffering based on existing technology;

[0013] Figure 3 This is a schematic diagram of a circular buffer based on existing technology;

[0014] Figure 4 This is a hardware structure block diagram of the data processing method according to an embodiment of this application;

[0015] Figure 5This is a flowchart of a data processing method according to an embodiment of this application;

[0016] Figure 6 This is a schematic diagram of bus scheduling according to an embodiment of this application;

[0017] Figure 7 This is a schematic diagram illustrating expansion or reduction according to an embodiment of this application;

[0018] Figure 8 This is a schematic diagram of a data processing apparatus according to an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0020] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] Figure 1 This is a schematic diagram of a single buffer based on existing technology, such as... Figure 1 As shown, buff-s is the source buffer, buff-d is the target buffer, and the arrow direction indicates the DMA transfer direction. In a single-buffered scheme, if the target buffer (buff-d) is not full or is being written, the system cannot operate on it and must wait for the write, resulting in a waste of system CPU resources. That is, in the traditional single-buffered DMA transfer method, data is written to a fixed buffer during the DMA transfer process. Although this method is simple, it has obvious problems when handling high data transfer rates or large amounts of data:

[0023] 1) Inefficient use of bus resources: During DMA transfer, the system bus is monopolized until the buffer is full or the transfer is completed. During this period, other devices cannot use the bus, resulting in low bus utilization.

[0024] 2) Risk of data backlog and loss: When data processing tasks have high real-time requirements, the serial operation of data processing and transmission in single buffer mode can easily cause data backlog in the buffer, and even data loss when the buffer is full.

[0025] 3) CPU resource waste: If the target cache is not full or is in the process of being written, the system needs to wait, during which time the CPU resources may be idle, resulting in resource waste.

[0026] To address the aforementioned issues, double buffering and circular buffering mechanisms have been gradually introduced into the technical field. Figure 2 This is a schematic diagram of double buffering based on existing technology, such as... Figure 2 As shown, the buffer structure is similar to that of a single buffer, except that it includes an additional target cache. Therefore, compared to a single buffer design, a double buffer design can simultaneously read data from the other target cache while writing to target cache 1 or target cache 2, achieving a balance between data computation and data transfer. Figure 3 This is a schematic diagram of a circular buffer based on existing technology, such as... Figure 3 As shown, once the current cache is full, the write address is adjusted to the next cache. Cache addresses are concatenated, allowing the CPU to read data from the first address and the DMA to transfer data based on the write address. Double buffering, by setting up two independent buffers, allows data transfer and processing to occur in parallel, thereby improving bus resource utilization and data processing speed. Circular buffering further optimizes buffer management by using a circular buffer structure and dynamic read / write pointers to achieve continuous and efficient data transfer, reducing buffer switching latency.

[0027] Although double buffering and circular buffering techniques have improved the efficiency and stability of data transmission to some extent, existing implementations still need improvement in the following aspects:

[0028] 1) Buffer management mechanism: Existing technologies may be inefficient in the automatic management and switching of buffers, especially when there are large fluctuations in data traffic. Buffer switching may not be timely, leading to data processing delays or data overflow.

[0029] 2) Bus resource allocation: In a system where multiple devices share a bus, existing technologies may not be able to effectively and dynamically adjust the allocation of bus resources, resulting in uneven resource allocation and low transmission efficiency for some devices.

[0030] 3) Compatibility with different devices: Existing technologies may not be fully compatible with all types of devices when implementing DMA double buffering and circular buffering, especially for devices with special data transmission requirements, which may require additional configuration and optimization.

[0031] 4) Dynamic adaptability: Existing technologies may lack the ability to respond and adapt quickly when faced with sudden changes in data transmission rate or volume, leading to a decline in system performance.

[0032] The specific application environment architecture or specific hardware architecture on which the execution of the data processing method depends is described here.

[0033] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 4 This is a hardware structure block diagram of the data processing method according to an embodiment of this application, such as... Figure 4 As shown, the server device may include one or more ( Figure 4 Only one is shown in the diagram. A processor 402 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 404 for storing data are also shown. The server device may further include a transmission device 406 for communication functions and an input / output device 408. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.

[0034] The memory 404 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data processing method in this embodiment. The processor 402 executes various functional applications and data processing by running the computer program stored in the memory 404, thus implementing the aforementioned method. The memory 404 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 404 may further include memory remotely located relative to the processor 402, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0035] Transmission device 406 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, transmission device 406 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0036] The embodiments of this application provide a data processing method, and the method is described in detail below in conjunction with the execution flow of the data processing method.

[0037] The following explains the technical terms used in this application:

[0038] DMA: Direct Memory Access (DMA) is a technology that allows peripherals or subsystems to exchange data directly with memory in a computer system without the intervention of the central processing unit (CPU). This mechanism significantly improves system efficiency, especially in scenarios involving high-speed data transfer.

[0039] DB: Data bus, the channel for data transfer between the CPU and memory or other devices. The width of the data bus determines the data transfer speed between the CPU and the outside world. Each transmission line can only transmit 1 bit of binary data at a time.

[0040] AB: Address bus. The CPU uses the address bus to specify memory locations. The address bus determines the maximum size of the memory space that the CPU can access.

[0041] CB: Control Bus. The CPU controls external devices through the control bus. The width of the control bus determines the CPU's control capability over external devices.

[0042] BUS: Bus. The CPU bus generally refers to the common connection line between the CPU and the chipset, also called the front-side bus (FSB). Whether it's a bus or a local bus, an internal bus or an external bus, they are all proposed to maximize the overall efficiency of a computer. You can think of them as the main roads and general roads in a city.

[0043] According to an embodiment of this application, a method embodiment for data processing is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] Figure 5 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 5 As shown, the method includes the following steps:

[0045] Step S502: The first transmission data is received alternately using the first target buffer and the second target buffer, wherein the first transmission data is data read from the source buffer.

[0046] Optionally, the first transfer data here refers to the data read from the source buffer via DMA, which typically includes raw data acquired from external devices (such as hard drives, network interfaces) or sensors.

[0047] In this embodiment, two target buffers, namely a first target buffer and a second target buffer, can be used to alternately receive the first transfer data read from the source buffer. This means that during DMA transfer, while one target buffer is writing data, the other buffer can be used by the system to process the previously written data, thereby achieving parallel operation of data transfer and processing.

[0048] Step S504: Transfer the second transmission data to the circular buffer, wherein the second transmission data is data obtained by preprocessing the first transmission data.

[0049] Optionally, a circular buffer is a circular data storage area that manages read and write operations through read and write pointers. This structure allows data to be written even when the buffer is full without overwriting existing data, until the read pointer catches up with the write pointer.

[0050] The second transmitted data is obtained after preprocessing the first transmitted data. Preprocessing may include data format conversion, filtering, grouping, and other operations to make the data more suitable for subsequent processing. Then, this preprocessed data is transferred to a circular buffer. The circular buffer provides continuous data access, avoiding the frequent switching and overhead of traditional linear buffers during data processing.

[0051] Step S506: Determine the scheduling method of the second transmission data based on the data characteristics of the first transmission data, wherein the scheduling method is one of the following: fixed scheduling method and dynamic scheduling method.

[0052] Optionally, in business scenarios with fixed data processing cycles, the system can periodically read data from the circular buffer for processing. This fixed scheduling method provides predictability and stability.

[0053] For scenarios with variable data processing cycles or high real-time requirements, a dynamic scheduling approach can be adopted to dynamically adjust the reading frequency and length based on the actual data transmission rate and demand, ensuring the real-time performance and efficiency of data processing.

[0054] In this embodiment, the scheduling method for the second transmitted data can be determined, including a fixed scheduling method and a dynamic scheduling method. The fixed scheduling method is suitable for scenarios with a fixed data processing cycle, while the dynamic scheduling method adjusts according to the fluctuations in the data processing cycle to adapt to data demands with non-fixed cycles.

[0055] Step S508: Read and process the second transmission data from the circular buffer according to the scheduling method.

[0056] In this embodiment, the second transmission data can be read from and processed according to the determined scheduling method. This ensures that data can be processed at an optimal pace, reducing waiting time and data backlog, and improving the system's response speed and data processing efficiency.

[0057] As described above, in this embodiment, the first transmission data can be received alternately using a first target buffer and a second target buffer, wherein the first transmission data is data read from the source buffer; the second transmission data is transferred to a circular buffer, wherein the second transmission data is data obtained by preprocessing the first transmission data; the scheduling method for the second transmission data is determined, wherein the scheduling method is one of the following: a fixed scheduling method and a dynamic scheduling method; the second transmission data is read from and processed from the circular buffer according to the scheduling method. Through key steps such as alternately using the target buffer to receive data, preprocessing and then transferring the data to the circular buffer, and flexibly selecting the scheduling method according to the business scenario, efficient parallelism of data transmission and processing is achieved, reducing the resource consumption of the system bus and improving the real-time performance and stability of data processing. It is particularly suitable for complex and ever-changing data-intensive applications, such as real-time data stream processing, image recognition, and network communication scenarios, and can significantly improve system performance and resource utilization efficiency.

[0058] Therefore, the technical solutions provided by the embodiments of this application solve the problem in the related technologies that the traditional DMA transmission method occupies bus resources for a long time in high data transmission rate environments, especially in single buffer mode, where bus resources are monopolized by a single DMA operation for a long time, which seriously restricts other devices from accessing the bus and reduces the overall utilization of the bus.

[0059] According to the above embodiments of this application, determining the scheduling mode of the second transmission data based on the data characteristics of the first transmission data includes: when the data characteristics indicate that the data processing period of the first transmission data is a fixed period, determining the scheduling mode as a fixed scheduling mode; and when the data characteristics indicate that the data processing period is a non-fixed period, determining the scheduling mode as a dynamic scheduling mode.

[0060] In this embodiment, two bus scheduling strategies are designed based on the business scenario: 1) Fixed-period scheduling. Fixed-period scheduling means that the system task periodically reads data from the circular buffer for processing. When there is data in the circular buffer, a specified length of data is read; if there is no data, no business operation is performed. This scheduling strategy is suitable for situations where business updates are fixed and the accuracy requirements for business response are high. 2) Dynamic adjustment scheduling. Dynamic scheduling requires setting configuration items [scheduling period, expansion threshold, expansion frequency, shrinkage threshold, shrinkage frequency], where the scheduling period refers to the system task processing cycle, for example, executing a data processing task once every 1 second.

[0061] When a circular buffer is at risk of overflow, the pressure on the buffer can be reduced by increasing the data read speed, for example, increasing the read frequency from 10 times per second to 20 times per second. Alternatively, if the system has available resources, expansion measures can be taken before overflow occurs, such as increasing the size of the circular buffer from 1MB to 2MB to increase data storage capacity.

[0062] By dynamically adjusting the read speed or capacity of the circular buffer, the problem of mismatch between data processing speed and data transmission speed is solved, ensuring that the system can maintain normal operation and timely data processing even when data traffic surges.

[0063] This approach enables the system to better adapt to changes in data transmission rates. By adjusting the read speed or scaling strategies, it improves the system's responsiveness to sudden data streams, reduces the risk of system downtime, and ensures smooth data transmission and timely processing.

[0064] According to the above embodiments of the present invention, determining the scheduling mode of the second transmission data based on the data characteristics of the first transmission data includes: determining the scheduling mode as a fixed scheduling mode when the data characteristics indicate that the data transmission fluctuation of the first transmission data is less than the fluctuation threshold; and determining the scheduling mode as a dynamic scheduling mode when the data characteristics indicate that the data transmission fluctuation is not less than the fluctuation threshold.

[0065] In this embodiment, when the data characteristics of the first transmitted data indicate that the system data is periodic and the transmission volume is small, that is, when the system data fluctuation is not large, the data scheduling method is determined to be a fixed scheduling method; conversely, if the data fluctuation is large, for example, when the system data is explosive, the data scheduling method is determined to be a dynamic scheduling method.

[0066] In fields such as network communication and real-time signal processing, data traffic is often unpredictable and volatile. Dynamic scheduling can flexibly adjust the priority and time allocation of DMA transfers according to real-time data traffic demands, avoiding bus congestion caused by fixed resource allocation when data volume is large, while reducing resource waste when data volume is small.

[0067] According to the above embodiments of this application, reading second transmission data from a circular buffer according to a scheduling method includes: when the scheduling method is a fixed scheduling method, periodically reading a specified length of second transmission data from the circular buffer according to a predetermined period; when the scheduling method is a dynamic scheduling method, generating a data reading method for the second transmission data, and reading and processing the second transmission data from the circular buffer according to the data reading method, wherein the data reading method includes: data reading frequency and single data reading length.

[0068] Choose the appropriate scheduling method based on the characteristics of the business scenario. For example, in video conferencing software, if the data transmission cycle is relatively fixed (e.g., 30 frames / second), a fixed scheduling method can be selected, processing data according to the principle of reading a certain amount of data per second. However, in an online gaming environment, data demand may fluctuate significantly. In this case, a dynamic scheduling method is adopted, adjusting the reading frequency and the length of each data read in real time according to the actual data transmission situation to ensure the real-time performance and efficiency of data processing.

[0069] By distinguishing between fixed and dynamic scheduling strategies, this method addresses the differences in data processing needs across various business scenarios, providing customized data processing rhythms for different scenarios and ensuring the flexibility and efficiency of data processing.

[0070] The flexible application of this scheduling strategy not only improves the real-time response capability of data processing, but also reduces the processing latency caused by the mismatch between data transmission rate and processing rate, thereby improving the user experience quality and the overall performance level of the system in various application scenarios.

[0071] According to the above embodiments of this application, the data reading method for generating the second transmission data includes: generating the data reading frequency and single data reading length of the second transmission data based on the data change frequency of the business scenario corresponding to the first transmission data or the second transmission data.

[0072] In this embodiment, for different business scenarios, the system can automatically generate the data reading frequency and single data reading length of the second transmitted data based on the changing frequency of the first transmitted data or the second transmitted data (preprocessed data), to ensure precise matching between data processing and data transmission. For example, if the business scenario involves the acquisition and processing of attitude data from a UAV, considering that the sampling frequency of gyroscope sensors is usually high, such as 10kHz, while the flight control system may read the preprocessed attitude data at a lower but fixed frequency, such as 100Hz, for control decisions, the system will generate an appropriate data reading frequency and single reading length based on the changing frequency of the sensor data and the reading requirements of the flight control system. This achieves effective data transmission and processing while avoiding data backlog and resource waste.

[0073] For example, firstly, the system analyzes the frequency of data changes in the business scenario. This involves monitoring the generation rate of source data and the update frequency of preprocessed data to assess the volatility and stability of the data stream, thus completing the data change frequency analysis. Next, based on the data change frequency and business requirements, the system calculates a reasonable data reading frequency to ensure that the data processing module can obtain the latest data in a timely manner without unnecessarily increasing the system load due to excessively frequent readings, achieving a match between reading requirements and actual needs. To balance data transmission efficiency and system processing real-time performance, the system also needs to determine the length of data read from the circular buffer each time. This step typically considers the processing capacity of the data processing unit and the size of the data transmission to find an optimal value that fully utilizes system resources without causing data processing latency.

[0074] The introduction of this mechanism not only improves the real-time performance and accuracy of data processing, but also dynamically adjusts the reading strategy according to the needs of actual business scenarios. This avoids resource waste caused by overly frequent readings and data latency caused by low-frequency readings, thereby minimizing the consumption of system bus resources while ensuring system performance. Furthermore, it enhances the system's flexibility and adaptability, enabling it to cope with changing data requirements in different types of business scenarios. For example, in video surveillance scenarios, it adjusts reading parameters based on camera resolution and frame rate; or in network communication scenarios, it optimizes the reading strategy based on network bandwidth and data packet size, further improving data processing efficiency and overall system performance.

[0075] According to the above embodiments of this application, the data processing method further includes: during the process of transferring the second transmission data to the circular buffer, monitoring the usage rate of the circular buffer and obtaining a monitoring result; when the monitoring result indicates that the circular buffer overflows after a predetermined period of time, performing an overflow recovery mechanism on the circular buffer, wherein overflow indicates that the write pointer of the circular buffer catches up with the read pointer and the circular buffer is full.

[0076] In this embodiment, the circular buffer structure is a ring-shaped data storage area, managed by two pointers (a read pointer and a write pointer). The read pointer points to the currently readable data position, and the write pointer points to the currently writable data position. When the write pointer reaches the end of the buffer, if the buffer is not full, it automatically returns to the beginning of the buffer to continue writing, creating a circular writing effect; the read pointer works similarly.

[0077] During DMA transfer, data is continuously written to a circular buffer sequentially. When data becomes available in the buffer, the system starts reading from the read pointer position. After reading, the read pointer moves forward one data unit. If the read pointer catches up with the write pointer, the buffer is empty; if the write pointer catches up with the read pointer, the buffer is full. This circular read-write mechanism enables continuous data transfer and processing, avoiding the overhead of frequent data movement required by traditional linear buffers during data processing, and further reducing bus occupancy.

[0078] According to the above embodiments of this application, when the monitoring results indicate that the circular buffer overflows after a predetermined period of time, an overflow recovery mechanism is executed on the circular buffer, including: increasing the reading speed of the second transmission data in the circular buffer.

[0079] In this embodiment, the usage rate of the circular buffer is continuously monitored during the data transfer process. If the circular buffer is detected to overflow within a certain period of time (e.g., 5 seconds), that is, the data write rate exceeds the read rate, causing the write pointer to catch up with the read pointer and the buffer to be full, the system will take measures to accelerate data reading to alleviate the overflow situation.

[0080] To address the overflow issue caused by a full data buffer, this approach monitors usage and adjusts management strategies accordingly, preventing data loss and processing delays, and improving system robustness and data security.

[0081] By automatically adjusting the read speed, the circular buffer can better match the fluctuations in the data stream, thereby reducing the risk of overflow and improving system stability and the continuity of data processing while maintaining data integrity and real-time performance.

[0082] According to the above embodiments of this application, when the monitoring results indicate that the circular buffer overflows after a predetermined period of time, an overflow recovery mechanism is executed on the circular buffer, including: expanding the circular buffer when there are free resources.

[0083] In this embodiment, to prevent the circular buffer from overflowing (i.e., the write pointer catches up with the read pointer and the buffer is full), this application sets up an overflow detection and handling mechanism. When an impending buffer overflow is detected, the system can take various actions, such as speeding up data reading (enhancing the processing capacity of the data processing module) or dynamically adjusting the buffer size (temporarily expanding the buffer capacity if system resources allow), to ensure data integrity and transmission stability.

[0084] By automatically adjusting the buffer capacity, the circular buffer can better match the fluctuations in the data stream, thereby reducing the risk of overflow while maintaining data integrity and real-time performance, and improving system stability and the continuity of data processing.

[0085] It should be noted that, in the embodiments of this application, double buffers and circular buffers are two different data buffering mechanisms. They play complementary roles in data processing and transmission and can be used in combination to optimize system performance and resource utilization. The relationship between double buffers and circular buffers and their synergistic effect in the system are described in detail below.

[0086] First, a dual-buffer system typically consists of two buffers of the same size, designated buffer A and buffer B. During data transfer, data is first written to one buffer (e.g., buffer A), while the other buffer (e.g., buffer B) can be read and processed by the CPU or other processors, enabling parallel data transfer and processing. When buffer A is full, the DMA controller automatically switches to buffer B for writing, while the CPU begins processing data in buffer A. This approach reduces waiting time and improves system throughput.

[0087] A circular buffer is a type of buffer with its head and tail connected, typically used to achieve more efficient data stream processing. A circular buffer consists of multiple contiguous or non-contiguous data buffer nodes, each with a fixed size. Through read / write pointer management, data can be written to the buffer continuously. When the write pointer reaches the end of the buffer, it automatically wraps back to the beginning and continues writing, creating a circular write effect. Circular buffers effectively avoid the overhead of frequent data copying between external buffers and the CPU. Furthermore, by dynamically adjusting the read / write pointers, they can balance data writing and reading, preventing data overflow or idle data.

[0088] Through the embodiments provided in this application, double buffers and circular buffers can be used in combination in advanced management of data transmission and processing to further improve the system's flexibility and response speed.

[0089] For example, during data writing and preprocessing, a dual buffer first receives data, and DMA can switch between buffers A and B for writing. Meanwhile, the CPU can simultaneously read and preprocess data in the buffers, transferring it to a circular buffer. This allows data transfer and preprocessing to occur synchronously, reducing waiting time. In efficient data stream processing, data transferred to the circular buffer can be continuously read and processed by the CPU or other processing units. The circular buffer dynamically adjusts read / write pointers to balance data writing and reading, reducing latency and bus resource consumption during data transfer.

[0090] For load balancing and resource optimization: Combining double buffers and circular buffers can achieve better load balancing. By dynamically adjusting the switching timing of double buffers and the read / write pointers of circular buffers, bus resource allocation can be optimized, avoiding data overflow or CPU idle waiting, thereby improving the overall utilization of system resources.

[0091] By combining the parallel transmission capabilities of a double buffer with the efficient data stream processing mechanism of a circular buffer, the system can achieve rapid data transmission, preprocessing, and load balancing when handling large amounts of continuous data, significantly improving system performance and resource utilization efficiency. In practical applications, this combination is typically used in scenarios requiring high-speed data transmission and real-time data processing, such as real-time video processing, network packet analysis, and sensor data stream processing.

[0092] According to the above embodiments of this application, the data processing method further includes: obtaining the usage rate, expansion threshold, and shrinkage threshold of the circular buffer; and triggering a dynamic adjustment mechanism for the circular buffer when the usage rate exceeds the expansion threshold or is less than the shrinkage threshold, so as to perform expansion or shrinkage operations on the circular buffer.

[0093] Optionally, the expansion threshold refers to the buffer adjustment strategy when the circular buffer usage exceeds a certain threshold range; the expansion frequency refers to increasing or decreasing the circular buffer cache after continuously meeting the adjustment threshold, and the same applies to shrinking. For example, setting the configuration option [1s, 80%, 5, 60%, 10] means that if the circular buffer usage exceeds 80% for 5 consecutive job cycles, it will be expanded; similarly, if the circular buffer usage is less than 60% for 10 consecutive job cycles, it will be shrunk.

[0094] In this embodiment, the usage rate, expansion threshold, and shrinkage threshold of the circular buffer can be obtained, and then it can be determined whether the circular buffer needs to be dynamically adjusted based on these data.

[0095] The settings for scheduling cycle, expansion threshold, expansion frequency, shrinkage threshold, and shrinkage frequency are as follows: 1) Scheduling cycle setting: Considering data traffic fluctuations: The scheduling cycle should be set based on the typical frequency of data traffic changes. If data traffic changes very quickly, a shorter scheduling cycle may be needed to respond to changes promptly; conversely, a longer cycle can be set to reduce the system overhead caused by scheduling. Balancing response and overhead: A scheduling cycle that is too short may lead to frequent resource checks, increasing the system load; a cycle that is too long may lead to a slow response to changes in resource demand. Therefore, ideally, a time interval that can both respond quickly and maintain low overhead should be found. 2) Setting expansion and shrinkage thresholds: Understanding workload: The expansion threshold should be set above the high utilization rate that the system frequently reaches to ensure sufficient buffer space when data surges. The shrinkage threshold should be below the low utilization rate to avoid excessive buffer resources being idle and wasted under low load. Allowing for margin: To avoid frequent expansion and contraction of the buffer, there should be a certain gap between these two thresholds. For example, if the expansion threshold is set to 80%, the shrinkage threshold might be set to 60%, so that even if the data volume fluctuates, adjustments won't be triggered immediately. 3) Setting the expansion and shrinkage frequencies: Avoid frequent adjustments: The expansion and shrinkage frequencies should be high enough to prevent the system from reacting to small fluctuations in data traffic. If the buffer utilization occasionally touches the threshold, there's no need to adjust immediately, as this may be a temporary phenomenon. Adapt to long-term trends: The frequency should be sufficient to reflect the long-term changing trends in data processing needs. If the data volume continues to grow for more than a period of time, the expansion frequency can be set to 5, meaning adjustments will only be made after the utilization exceeds the threshold for 5 consecutive scheduling cycles; similarly, if the data volume continues to decrease, the shrinkage frequency can be set to 10, meaning the buffer will only shrink after the utilization is below the threshold for 10 consecutive cycles.

[0096] For example, in a real-time data analysis system, the system needs to provide sufficient buffer space when the data volume is large, while avoiding resource waste when the data volume is small. Here is a possible configuration setting example: Scheduling cycle: Set to 500 milliseconds for quick response to slight changes in data traffic, without causing excessive system overhead. Expansion threshold: Set to 85% to ensure sufficient buffer space to prevent data overflow during peak data periods. Expansion frequency: Set to 3, meaning that an expansion operation is performed when the average buffer utilization exceeds 85% for three consecutive scheduling cycles. Shrinkage threshold: Set to 55%, below which the system begins to consider releasing buffer resources to save memory. Shrinkage frequency: Set to 6, meaning that a shrinkage operation is only performed when the buffer utilization is below 55% for six consecutive cycles.

[0097] According to the above embodiments of this application, a dynamic adjustment mechanism for triggering a circular buffer is provided to perform expansion or contraction operations on the circular buffer, including: performing expansion operations on the circular buffer when the number of times the usage rate exceeds the expansion threshold is the expansion frequency; and performing contraction operations on the circular buffer when the number of times the usage rate is less than the contraction threshold is the contraction frequency.

[0098] In this embodiment, the usage rate of the circular buffer is obtained and compared with preset expansion and contraction thresholds. If the usage rate exceeds 80% for 5 consecutive times (the specific value is adjusted according to the actual situation), the circular buffer is considered to need to be expanded, and an additional cache node is inserted before the read node. Conversely, if the usage rate is below 60% for 10 consecutive times, it indicates that the buffer capacity is excessive, and a cache node is deleted at the read node to reduce resource consumption.

[0099] By dynamically monitoring and adjusting the size of the circular buffer, this method solves the problem of mismatch between buffer capacity and actual data demand, avoiding both resource waste and data backlog.

[0100] The dynamic adjustment mechanism enables the circular buffer to automatically expand or shrink according to the actual amount of data. This intelligent resource management method improves the efficiency of data processing, optimizes the utilization of the bus and other resources, and enhances the overall performance and stability of the system.

[0101] Figure 6 This is a schematic diagram of bus scheduling according to an embodiment of this application, such as... Figure 6 As shown, the size of the double-buffered buffer buff-d is the same as that of each buffer buff-c in the circular buffer. When data is available at the source (e.g., for I / O sampling), a DMA transfer is triggered, and the data is transferred from buff-s to the double-buffered buffer buff-d. When buff-d is full, a DMA target address full interrupt is triggered, switching the DMA transfer target address of the source buff-s, and simultaneously transferring the data that is full in buff-d to the circular buffer buff-c. The circular buffer buff-c is designed as follows: Figure 6 As shown, it is a doubly linked list. A configuration file [list size, read node, write node] records the list read / write information of the circular buffer, and node configuration [node address, node size, previous node address, next node address] records node information. When writing to the list, the write node moves to the next node; after reading, the read node moves to the next node synchronously.

[0102] According to the above embodiments of this application, when the usage rate exceeds the expansion threshold or the usage rate is less than the shrinkage threshold, a dynamic adjustment mechanism of the circular buffer is triggered to perform expansion or shrinkage operations on the circular buffer, including: when performing expansion operations on the circular buffer, inserting at least one node before the current read node of the circular buffer; when performing shrinkage operations on the circular buffer, deleting at least one node from the current read node of the circular buffer.

[0103] In this embodiment, when the usage rate of the circular buffer exceeds the expansion threshold (e.g., 80%) at a certain frequency (e.g., 5 times), the system will automatically trigger a buffer expansion operation; conversely, if the usage rate is below the shrinkage threshold (e.g., 60%) for a certain number of times (e.g., 10 times), a shrinkage operation will be triggered. The expansion operation involves inserting one or more cache nodes before the current read node to increase data storage space; while the shrinkage operation involves deleting the cache nodes before the read node to save system resources.

[0104] To address the issue of insufficient or excessive buffer capacity under static configuration, a dynamic adjustment mechanism based on utilization rate is introduced, enabling the buffer capacity to be automatically adjusted according to the actual data transmission needs.

[0105] By expanding and shrinking the size, the circular buffer can dynamically adjust its size to better match the rapidly changing data transmission rate, ensuring the real-time performance and efficiency of data processing, while also avoiding excessive consumption or waste of resources.

[0106] Figure 7 This is a schematic diagram illustrating the expansion or reduction of capacity according to embodiments of this application, such as... Figure 7 As shown, the cyclic buffer utilization rate can be calculated. When the utilization rate is greater than the expansion threshold, the count is incremented by 1 continuously; conversely, when the utilization rate is less than the shrinkage threshold, the count is incremented by 1 continuously. Expansion or shrinkage is performed based on the above results.

[0107] In other words, when the continuous utilization rate reaches the expansion configuration, an expansion operation is performed. When the continuous utilization rate falls below the shrinkage threshold, a shrinkage operation is performed. The basic logic of the expansion operation is to insert a new node before the current read node. The basic logic of the shrinkage operation is to delete the node preceding the current read node.

[0108] In the application embodiment, the usage rate of the circular buffer is calculated as follows: ,in, Indicates the buffer cache utilization rate. This indicates the count of unused cache nodes. This indicates the total number of cache nodes.

[0109] The technical solutions provided in the above embodiments of this application can effectively optimize the mismatch between I / O and the system bus, achieving a balance between bus resources and I / O data, improving CPU utilization, and reducing CPU load on invalid data transfer ends. Through the comprehensive application of double buffering, circular buffering mechanisms, and dynamic scheduling strategies, it effectively solves problems such as low data transmission and processing efficiency, high bus resource consumption, and buffer capacity mismatch in existing technologies, significantly improving the system's data processing capabilities, bus resource utilization efficiency, and overall stability. Specific values ​​(e.g., thresholds and frequencies) in the embodiments can be flexibly set according to different business scenarios and hardware characteristics to achieve optimal performance. These technical solutions are particularly suitable for scenarios with high data throughput and real-time data processing requirements, such as big data analysis, real-time video stream processing, and high-performance computing, providing strong technical support for modern data-intensive applications.

[0110] Moreover, by optimizing data transmission, it can respond to the demand for high-frequency sampling. While ensuring data integrity and fast response, it effectively reduces the impact of frequent I / O processing on system load, achieving coordinated processing in scenarios where I / O and CPU capabilities are unbalanced, thus ensuring data security and system stability.

[0111] According to the above embodiments of this application, before alternately receiving the first transmitted data using the first target buffer and the second target buffer, the data processing method further includes: obtaining the data block capacity and data transmission requirements of the business scenario corresponding to the first transmitted data; determining the buffer capacity based on the data block capacity and data transmission requirements; and configuring the first target buffer and the second target buffer based on the buffer capacity.

[0112] Optionally, the aforementioned data transmission requirements may include, but are not limited to, data transmission duration and data transmission frequency. For example, two buffers of the same size can be set up, namely buffer A (first target buffer) and buffer B (second target buffer).

[0113] It should be noted that the size of each buffer can be dynamically configured according to the data block size and transmission frequency in the actual application scenario to ensure that data can be stored sufficiently without wasting memory resources.

[0114] For example, in UAV attitude control, the gyroscope sensor sends three-dimensional attitude data to the flight control chip at a frequency of 10MHz, and the flight control chip reads the data at a fixed period of 5ms. Therefore, assuming that the size of each sampled data is 2 bytes, the buffer size can be set to 50000*2, which equals 100,000 bytes.

[0115] In this embodiment, before performing double-buffered transmission, the data transmission requirements of the business scenario are analyzed. For example, in a video surveillance system, it may be necessary to estimate the data block capacity based on camera resolution and frame rate. Based on these analysis results, the sizes of buffer A and buffer B are dynamically configured to ensure that they can effectively carry the expected data traffic without excessively consuming system resources.

[0116] To address the buffer size mismatch issue caused by static configuration, a method for dynamically configuring the buffer size based on business scenarios is introduced. This avoids resource waste caused by an excessively large buffer and prevents overflow risks caused by an excessively small buffer.

[0117] The above dynamic buffer size configuration enhances the system's adaptability and resource optimization capabilities, reduces CPU idle waiting time, and also reduces the possibility of data loss.

[0118] Furthermore, when designing a DMA double-buffering mechanism, properly configuring the sizes of the first and second target buffers is crucial, as it directly affects the utilization of the system bus, data processing efficiency, and overall system performance. Based on data block capacity and data transfer frequency requirements, specific strategies for configuring the buffers can be as follows:

[0119] 1) Assess data transmission requirements: First, it is necessary to analyze the specific data transmission scenario to determine the average size of each data block and the frequency of data transmission. For example, in the attitude control of a UAV, the gyroscope sensor sends data to the flight control chip at a frequency of 10MHz, with each data transmission consisting of 2 bytes. This data forms the basis for the configuration buffer.

[0120] 2) Determine the buffer size: Based on the evaluation results, calculate the total amount of data to be transferred within one cycle (e.g., a 5ms cycle). This will determine the minimum buffer capacity. For example, if 50,000 data blocks need to be read every 5ms, and each data block is 2 bytes, then the buffer capacity should be at least 100,000 bytes. Typically, some redundancy is added on top of this to cope with sudden data transfer demands and to avoid buffer overflow.

[0121] 3) Dynamic Adjustment Mechanism: Considering that data transmission demands may fluctuate over time, a dynamic adjustment mechanism should be incorporated into the buffer design. When a change in data transmission frequency or data block size is detected, the buffer size should be adjusted immediately to adapt to the new demands. For example, if the data transmission frequency suddenly increases, the buffer size can be automatically increased, and vice versa, to maintain a balance between data buffering and DMA transfers.

[0122] 4) Consider system resource limitations: When configuring the buffer size, the system's memory resource limitations should also be considered. The buffer should not be set too large, consuming excessive memory resources and affecting the execution of other tasks. Therefore, a balance needs to be found between meeting data transfer requirements and avoiding wasting memory resources.

[0123] 5) Simulation and Testing: Before actual deployment, the configured strategy should be simulated and tested to verify whether the buffer size setting is reasonable. Testing should include scenarios with different data transmission frequencies and different data block sizes to ensure that the double buffering mechanism can operate efficiently and stably under various conditions.

[0124] 6) Feedback and Optimization: During system operation, the usage of the buffer and system performance feedback should be continuously monitored. If the buffer settings are found to cause a decrease in system performance or waste of resources, the buffer configuration should be adjusted in a timely manner and the DMA transfer strategy optimized.

[0125] By employing the above strategies, the first and second target buffers can be effectively configured to meet the requirements of data transmission frequency, adapt to changes in data block capacity, and take into account system resource limitations. This ensures that the DMA double buffering mechanism can operate efficiently and stably in various scenarios, reduce system bus consumption, and improve overall system performance.

[0126] According to the above embodiments of this application, the method of alternately receiving first transmission data using a first target buffer and a second target buffer includes: when a direct memory access operation is detected to be triggered, reading the first transmission data from the source buffer through direct memory access and writing the first transmission data into the first target buffer; and when the first target buffer is detected to be full, writing the first transmission data into the second target buffer.

[0127] In this embodiment, when a DMA operation is triggered, for example, when reading data from a hard disk or receiving data from a network, the data is first written to buffer A; when buffer A is full, the DMA operation automatically switches to buffer B, and the system begins to preprocess the data in buffer A.

[0128] In other words, when DMA begins transferring data, it first writes the data to buffer A. While writing to buffer A, the system can simultaneously process the transferred data in buffer B (data in buffer B is transferred to the circular buffer). When buffer A is full, DMA automatically switches to buffer B for data writing, while the system continues processing the data in buffer A. This alternating writing and processing method allows data transfer and processing to occur in parallel, reducing system latency and thus minimizing continuous bus occupancy.

[0129] To avoid blocking other DMA operations during data writing, a double buffering mechanism is used, allowing write and read operations to be performed alternately, thereby improving the parallelism of data processing without interrupting bus services.

[0130] This implementation ensures continuous data reading and processing by timely switching of the buffer, reduces the system bus occupancy time, and improves the stability and efficiency of data transmission.

[0131] According to the above embodiments of the present invention, transferring the second transmission data to a circular buffer includes: listening to a transfer event and obtaining a listening result, wherein the transfer event is used to transfer the second transmission data to a circular buffer; and transferring the second transmission data to a circular buffer when the listening result indicates that the transfer event has been triggered.

[0132] In this embodiment, the timing for transferring the preprocessed second transmission data from the double buffer to the circular buffer can be determined by monitoring the transfer event. The triggering methods for the transfer event can include various approaches.

[0133] For example, when the double buffer is full, a transfer event is triggered; that is, upon detecting that the double buffer is full, a transfer event is triggered to transfer the second transmitted data to the circular buffer. Alternatively, a buffer threshold can be set; when the buffered data in the double buffer reaches the threshold, a transfer event is triggered. Furthermore, a scheduled task can be set, for example, to trigger a transfer event every hour to transfer the second transmitted data to the circular buffer. Of course, other methods exist, but will not be elaborated upon here.

[0134] According to the above embodiments of this application, the data processing method further includes: during the process of writing the first transmission data into the first target buffer, obtaining the cache state of the second target buffer; when the cache state indicates that the second target buffer is in an idle state, preprocessing the cached data in the second target buffer.

[0135] In this embodiment, while data is being written to buffer A or B, the buffer status of the other buffer is detected. If the current buffer (e.g., B) is idle, i.e. not full, preprocessing of the data within it can begin; similarly, when buffer A begins receiving data, preprocessing of the data already buffered in buffer B can be performed. Here, preprocessing may involve steps such as data compression and format conversion.

[0136] The main issue here is how to make full use of idle buffers during buffer switching. By preprocessing the data in the idle buffers, waiting time is reduced, and the immediacy of data processing and resource utilization efficiency are improved.

[0137] The application of the above preprocessing mechanism further shortens the time interval between data reception, storage and final processing, improves system response speed, and helps to balance the allocation of system bus resources.

[0138] According to the above embodiments of this application, the data processing method further includes: preprocessing the cached data in the first target buffer during the process of writing the first transmission data into the second target buffer.

[0139] In DMA double-buffering, data transfer and processing can be performed in parallel to improve efficiency and reduce continuous occupancy of the system bus. The specific process is as follows:

[0140] 1) Initialize dual buffers: Before the DMA transfer begins, first initialize two buffers, labeled as buffer A and buffer B respectively, and ensure that they are both in an available state.

[0141] 2) DMA Start Transfer to Buffer A: When a DMA operation is triggered, the system first writes data to buffer A. During this period, the DMA operation runs independently of the CPU, directly writing data captured from the source device (such as a sensor, network interface, etc.) into buffer A until buffer A is full or a predetermined write trigger point is reached.

[0142] 3) Parallel Data Processing: While write operations are performed in buffer A, the system can process data in buffer B in parallel. This processing may include reading, converting, compressing, or directly transferring the data to a circular buffer. This is because when buffer A is receiving data, buffer B is idle and can be used for data processing, thus achieving parallel data transmission and processing.

[0143] 4) Buffer Switching: When buffer A is full, DMA will automatically switch to buffer B for data writing. At the same time, the system will start reading data from buffer A for processing or transferring it to the circular buffer. This switching process is usually seamless, ensuring the continuity of data transmission.

[0144] 5) Circular Buffer Transfer: Processed data is transferred to a circular buffer. A circular buffer is an efficient data storage structure that manages read and write operations through read and write pointers, preventing data overwriting, ensuring data continuity and integrity, and providing a continuous data stream for CPU processing.

[0145] 6) Continuous Loop: The above process continues to loop during system operation. DMA alternately transfers data between buffer A and buffer B, while the system processes the data in the two buffers in parallel, ensuring high efficiency in data transfer and processing.

[0146] Through the aforementioned mechanisms, the DMA double-buffering and circular buffering methods can effectively reduce system bus occupancy time, improve the real-time performance and efficiency of data transmission and processing, and ensure system stability and data integrity. This design is particularly suitable for applications with large data volumes and high transmission speed requirements, such as real-time signal processing, high-speed network communication, or multimedia data stream processing.

[0147] Furthermore, in this embodiment of the application, initializing the buffers is the first step in ensuring a smooth and efficient data transfer process in the DMA double-buffering mechanism. The following are the general steps for initializing two buffers A and B:

[0148] 1) Memory Allocation: The system first needs to allocate sufficient memory space for buffers A and B. The required memory capacity for each buffer is determined based on the expected data transfer rate, data block size, and system memory resources. For example, if 100MB of data is expected to be transferred per second, and the system can accept a buffer size of 50MB, then 50MB of memory space can be allocated to each buffer.

[0149] 2) Address Mapping: After memory is allocated, the physical addresses of these buffers need to be mapped to an address space recognizable by the DMA controller. This allows the DMA controller to directly access these buffers without CPU intervention. This typically involves creating DMA descriptors or configuring the DMA controller's registers to indicate the source and destination buffer locations of the data.

[0150] 3) Setting Initial State: Buffer initialization also includes setting their initial state. For example, buffer A is set as the buffer currently receiving data, while buffer B is temporarily empty or being processed by the CPU. This means that during initialization, the DMA controller should point to buffer A for data writing.

[0151] 4) Configure buffer size and boundaries: Define the size limits for each buffer, telling the DMA controller when to stop writing to the current buffer and switch to another buffer. This can be done by setting specific interrupt points, i.e., triggering a DMA interrupt when the buffer reaches a certain threshold, thereby switching to another buffer.

[0152] 5) Initialize read and write pointers: For buffer B, it is crucial to initialize the read and write pointers before it is used for data processing. The read pointer should point to the starting address of buffer B, while the write pointer is initialized to point to the starting address or a later address, depending on the buffer's prefilling strategy.

[0153] 6) Setting up interrupt and event handlers: During buffer initialization, corresponding interrupt and event handlers also need to be set up to handle buffer switching, data processing completion, and any potential error conditions. For example, when buffer A is full, the system should be able to trigger an interrupt, notify the DMA controller to switch to buffer B, and start the event handler for data processing in buffer A.

[0154] 7) Checking and Verification: After initialization, a series of checks and verifications are performed to ensure that the buffer address is correct, the size is as expected, the read / write pointers are initialized correctly, and the interrupt handler functions properly. This step is crucial for finding and correcting any errors during the initialization phase.

[0155] Through the above steps, the two buffers A and B are initialized and ready to receive and process data. This initialization strategy is the foundation for the successful operation of the DMA double-buffering mechanism, ensuring the continuity and efficiency of data transmission while reducing the CPU load and system bus usage time.

[0156] Therefore, the technical solution provided in this application has the following beneficial effects: 1) Reduced bus consumption: Through double buffering and circular buffering mechanisms, parallel operation of data transmission and processing is realized, reducing the continuous occupation time of the system bus by DMA transmission, improving the utilization rate of bus resources, providing more opportunities for bus access by other devices, reducing the risk of data backlog and transmission interruption caused by data processing delays, enhancing the stability of the system in complex data transmission environments, and thus improving the data transmission capability and performance of the entire system. 2) Improved system performance: Reduced data transmission delay and waiting time, accelerated data processing speed, enabling the system to run more efficiently and stably when processing large amounts of data. Whether in application scenarios such as file transfer, network communication, or multimedia processing, it can significantly improve the system's response speed and throughput, providing users with a better user experience. 3) Dynamic adjustment strategy: Using configuration items and combined with circular buffering characteristics, the circular buffer is dynamically expanded / shrunken, reducing the occupation of system resources.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0158] According to an embodiment of this application, a data processing apparatus for implementing the above-described data processing method is also provided. Figure 8 This is a schematic diagram of a data processing apparatus according to an embodiment of this application, such as... Figure 8As shown, the device includes: a receiving unit 801, a transfer unit 803, a first determining unit 805, and a processing unit 807. The device will be described below.

[0159] The receiving unit 801 is used to alternately receive first transmission data using a first target buffer and a second target buffer, wherein the first transmission data is data read from the source buffer.

[0160] The transfer unit 803 is used to transfer the second transmission data to the circular buffer, wherein the second transmission data is data obtained by preprocessing the first transmission data.

[0161] The first determining unit 805 is used to determine the scheduling mode of the second transmission data based on the data characteristics of the first transmission data, wherein the scheduling mode is one of the following: fixed scheduling mode and dynamic scheduling mode.

[0162] Processing unit 807 is used to read and process the second transmission data from the circular buffer according to the scheduling method.

[0163] It should be noted that the receiving unit 801, the transfer unit 803, the first determining unit 805, and the processing unit 807 mentioned above correspond to steps S502 to S508 in the above embodiments. The four units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0164] As can be seen from the above, in the scheme described in the above embodiments of this application, the receiving unit can alternately receive the first transmission data using the first target buffer and the second target buffer, wherein the first transmission data is data read from the source buffer; the transfer unit can transfer the second transmission data to the circular buffer, wherein the second transmission data is data obtained by preprocessing the first transmission data; the first determining unit is used to determine the scheduling method of the second transmission data, wherein the scheduling method is one of the following: fixed scheduling method and dynamic scheduling method; the processing unit reads and processes the second transmission data from the circular buffer according to the scheduling method. Through key steps such as alternately using the target buffer to receive data, preprocessing and then transferring to the circular buffer, and flexibly selecting the scheduling method according to the business scenario, efficient parallel transmission and processing of data is achieved, reducing the resource consumption of the system bus and improving the real-time performance and stability of data processing. It is especially suitable for complex and ever-changing data-intensive applications, such as real-time data stream processing, image recognition, network communication, etc., and can significantly improve system performance and resource utilization efficiency.

[0165] Therefore, the technical solutions provided by the embodiments of this application solve the problem in the related technologies that the traditional DMA transmission method occupies bus resources for a long time in high data transmission rate environments, especially in single buffer mode, where bus resources are monopolized by a single DMA operation for a long time, which seriously restricts other devices from accessing the bus and reduces the overall utilization of the bus.

[0166] Optionally, the first determining unit includes: a first determining module, used to determine the scheduling method as a fixed scheduling method when the data processing cycle of the business scenario is a fixed cycle, wherein the business scenario is the scenario corresponding to the first transmitted data or the second transmitted data; and a second determining module, used to determine the scheduling method as a dynamic scheduling method when the data processing cycle of the business scenario is a non-fixed cycle.

[0167] The first determining unit includes: a third determining module, configured to determine the scheduling mode as the fixed scheduling mode when the data characteristics indicate that the data transmission fluctuation of the first transmitted data is less than the fluctuation threshold; and a fourth determining module, configured to determine the scheduling mode as the dynamic scheduling mode when the data characteristics indicate that the data transmission fluctuation is not less than the fluctuation threshold.

[0168] Optionally, the processing unit includes: a reading module, configured to periodically read a specified length of second transmission data from a circular buffer according to a predetermined period when the scheduling mode is a fixed scheduling mode; and a generation module, configured to generate a data reading method for the second transmission data when the scheduling mode is a dynamic scheduling mode, and read and process the second transmission data from the circular buffer according to the data reading method, wherein the data reading method includes: data reading frequency and single data reading length.

[0169] Optionally, the generation module includes: a generation submodule, used to generate the data reading frequency and single data reading length of the second transmission data based on the data change frequency of the business scenario corresponding to the first transmission data or the second transmission data.

[0170] Optionally, the data processing device further includes: a monitoring unit, used to monitor the usage rate of the circular buffer during the process of transferring the second transmission data to the circular buffer, and obtain a monitoring result; and an execution unit, used to execute an overflow recovery mechanism for the circular buffer when the monitoring result indicates that the circular buffer overflows after a predetermined period of time, wherein overflow indicates that the write pointer of the circular buffer catches up with the read pointer and the circular buffer is full.

[0171] Optionally, the execution unit includes: an addition module for increasing the reading speed of the second transmission data in the circular buffer.

[0172] Optionally, the execution unit includes an execution module, configured to expand the circular buffer when idle resources are available.

[0173] Optionally, the data processing device further includes: a first acquisition unit, used to acquire the usage rate, expansion threshold, and shrinkage threshold of the circular buffer; and a triggering unit, used to trigger a dynamic adjustment mechanism of the circular buffer when the usage rate exceeds the expansion threshold or the usage rate is less than the shrinkage threshold, so as to perform expansion or shrinkage operations on the circular buffer.

[0174] Optionally, the triggering unit includes: an expansion module, configured to perform an expansion operation on the circular buffer when the number of times the usage rate exceeds the expansion threshold is the expansion frequency; and a shrinkage module, configured to perform a shrinkage operation on the circular buffer when the number of times the usage rate is less than the shrinkage threshold is the shrinkage frequency.

[0175] Optionally, the triggering unit includes: an insertion module, configured to insert at least one node before the current read node of the circular buffer when performing an expansion operation on the circular buffer; and a deletion module, configured to delete at least one node on the current read node of the circular buffer when performing a shrink operation on the circular buffer.

[0176] Optionally, the data processing apparatus further includes: a second acquisition unit, configured to acquire the data block capacity and data transmission requirements of the business scenario corresponding to the first transmission data before alternately receiving the first transmission data using the first target buffer and the second target buffer; a second determination unit, configured to determine the cache capacity based on the data block capacity and the data transmission requirements; and a configuration unit, configured to configure the first target buffer and the second target buffer based on the cache capacity.

[0177] The transfer unit includes: a listening module for listening to transfer events and obtaining listening results, wherein the transfer event is used to transfer the second transmission data to the circular buffer; and a transfer module for transferring the second transmission data to the circular buffer when the listening result indicates that the transfer event has been triggered.

[0178] Optionally, the receiving unit includes: a read / write module, configured to read first transmission data from the source buffer and write the first transmission data to the first target buffer via direct memory access when a direct memory access operation is detected to be triggered; and a monitoring module, configured to write the first transmission data to the second target buffer when the first target buffer is detected to be full.

[0179] Optionally, the data processing apparatus further includes: a third acquisition unit, configured to acquire the cache state of the second target buffer during the process of writing the first transmission data into the first target buffer; and a first preprocessing unit, configured to preprocess the cached data in the second target buffer when the cache state indicates that the second target buffer is in an idle state.

[0180] Optionally, the data processing apparatus further includes a second preprocessing unit, configured to preprocess the cached data in the first target buffer during the process of writing the first transmission data into the second target buffer.

[0181] According to another aspect of the embodiments of this application, an electronic device is also provided, including: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a data processing method for performing any of the above.

[0182] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein when the computer program is executed by a processor, it implements the steps of any of the above data processing methods.

[0183] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the steps of any of the above-described data processing methods.

[0184] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0185] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0186] The data processing method provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data processing method, characterized in that, include: The first transmission data is received alternately using a first target buffer and a second target buffer, wherein the first transmission data is data read from the source buffer; The second transmission data is transferred to a circular buffer, wherein the second transmission data is data obtained by preprocessing the first transmission data; The scheduling method for the second transmission data is determined based on the data characteristics of the first transmission data, wherein the scheduling method is one of the following: fixed scheduling method and dynamic scheduling method; The second transmission data is read from and processed according to the scheduling method described above; The process of reading the second transmission data from the circular buffer according to the aforementioned scheduling method includes: When the scheduling mode is the fixed scheduling mode, the second transmission data of a specified length is periodically read from the circular buffer according to a predetermined period. When the scheduling mode is the dynamic scheduling mode, a data reading mode for the second transmitted data is generated, and the second transmitted data is read from the circular buffer and processed according to the data reading mode, wherein the data reading mode includes: data reading frequency and single data reading length; The data processing method further includes: Obtain the usage rate, expansion threshold, and shrinkage threshold of the circular buffer; When the usage rate exceeds the expansion threshold or the usage rate is less than the shrinkage threshold, the dynamic adjustment mechanism of the circular buffer is triggered to perform an expansion or shrinkage operation on the circular buffer. The data processing method further includes: initializing the first target buffer and the second target buffer to ensure that the first target buffer and the second target buffer are in an available state.

2. The data processing method according to claim 1, characterized in that, Determining the scheduling method for the second transmission data based on the data characteristics of the first transmission data includes: When the data characteristics indicate that the data processing cycle of the first transmitted data is a fixed cycle, the scheduling method is determined to be the fixed scheduling method; When the data characteristics indicate that the data processing cycle is not a fixed cycle, the scheduling method is determined to be the dynamic scheduling method.

3. The data processing method according to claim 1, characterized in that, Determining the scheduling method for the second transmission data based on the data characteristics of the first transmission data includes: When the data characteristics indicate that the data transmission fluctuation of the first transmitted data is less than the fluctuation threshold, the scheduling method is determined to be the fixed scheduling method; When the data characteristics indicate that the data transmission fluctuation is not less than the fluctuation threshold, the scheduling method is determined to be the dynamic scheduling method.

4. The data processing method according to claim 1, characterized in that, The data reading methods for generating the second transmitted data include: The data reading frequency and the single data reading length of the second transmission data are generated based on the data change frequency of the first transmission data or the second transmission data.

5. The data processing method according to claim 1, characterized in that, The data processing method further includes: During the process of transferring the second transmitted data to the circular buffer, the usage rate of the circular buffer is monitored, and the monitoring results are obtained. When the monitoring results indicate that the circular buffer overflows after a predetermined period of time, an overflow recovery mechanism is executed on the circular buffer, wherein the overflow indicates that the write pointer of the circular buffer catches up with the read pointer and the circular buffer is full.

6. The data processing method according to claim 5, characterized in that, When the monitoring result indicates that the circular buffer overflows after a predetermined period of time, an overflow recovery mechanism is executed on the circular buffer, including: Increase the reading speed of the second transmitted data in the circular buffer.

7. The data processing method according to claim 5, characterized in that, When the monitoring result indicates that the circular buffer overflows after a predetermined period of time, an overflow recovery mechanism is executed on the circular buffer, including: When there are available idle resources, the circular buffer is expanded.

8. The data processing method according to claim 1, characterized in that, Triggering the dynamic adjustment mechanism of the circular buffer to perform expansion or contraction operations on the circular buffer includes: When the number of times the usage rate exceeds the expansion threshold is defined as the expansion frequency, the expansion operation is performed on the circular buffer. When the number of times the usage rate is less than the shrinkage threshold is the shrinkage frequency, the shrinkage operation is performed on the circular buffer.

9. The data processing method according to claim 1, characterized in that, When the usage rate exceeds the expansion threshold or the usage rate is less than the shrinkage threshold, the dynamic adjustment mechanism of the circular buffer is triggered to perform expansion or shrinkage operations on the circular buffer, including: When performing the expansion operation on the circular buffer, at least one node is inserted before the current read node of the circular buffer; When performing the shrinking operation on the circular buffer, at least one node is deleted from the current read node of the circular buffer.

10. The data processing method according to claim 1, characterized in that, The first transmitted data is received alternately using a first target buffer and a second target buffer, including: When a direct memory access operation is triggered, the first transmission data is read from the source buffer via the direct memory access method, and the first transmission data is written to the first target buffer. When the first target buffer is detected to be full, the first transmission data is written to the second target buffer.

11. The data processing method according to claim 10, characterized in that, The second transmitted data is transferred to the circular buffer, including: The transfer event is monitored to obtain the monitoring result, wherein the transfer event is used to transfer the second transmitted data to the circular buffer; When the listening result indicates that the transfer event has been triggered, the second transmitted data is transferred to the circular buffer.

12. The data processing method according to claim 10, characterized in that, The data processing method further includes: During the process of writing the first transmitted data into the second target buffer, the cached data in the first target buffer is preprocessed.

13. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing the data processing method according to any one of claims 1 to 12.

Images