Big data visualization downsampling method, computer equipment and storage medium
By performing data segmentation, zero-copy, and downsampling between the main thread and child threads, and utilizing Web Worker technology to process data independently, the low rendering efficiency and response latency issues of Echarts in big data scenarios are resolved, achieving efficient rendering and smooth interaction, and improving the system's response speed and stability.
Patent Information
- Application Number
- CN202511406018.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-29
- Publication Date
- 2026-02-03
AI Technical Summary
In big data-driven decision analysis scenarios, Echarts faces resource contention, page lag and response latency caused by improper memory management when processing millions of data points, affecting the system's response speed and stability, especially when scaling and interacting, task backlog and latency are severe.
By performing data chunking, zero-copying, and downsampling between the main thread and child threads, and using Web Worker technology to create independent child threads for data processing, combined with binary search and streaming return of downsampling results, efficient processing of rendering and scaling operations is achieved.
It achieves high-performance rendering and smooth interaction in real-time visualization scenarios with millions of data points, significantly improving the system's response speed and stability, avoiding main thread blocking and frequent garbage collection, and ensuring smooth front-end page response.
Smart Images
Figure CN121456047A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a downsampling method for big data visualization, a computer device, and a computer-readable storage medium. Background Technology
[0002] In big data-driven decision analysis scenarios, data visualization, as a core tool for information value mining, directly impacts the efficiency of business insights due to its performance bottlenecks. While Echarts, an open-source visualization library based on JavaScript, performs excellently in typical scenarios, it faces numerous challenges when processing millions of data points. For example, resource contention in a single-threaded architecture can cause the main front-end thread to block as it must simultaneously handle data computation and UI rendering tasks. In terms of memory management, injecting large amounts of data directly into memory can trigger garbage collection, causing page lag. These factors all contribute to low front-end rendering efficiency and affect system responsiveness and stability.
[0003] Furthermore, during scaling interactions, all data needs to be recalculated and rendered, which can easily lead to task backlog and response delays. This not only affects the user experience but also further impacts the system's response speed and stability.
[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention
[0005] The main purpose of this application is to provide a downsampling method, computer device, and computer-readable storage medium for big data visualization, aiming to achieve efficient rendering and smooth interaction of big data visualization, and improve the system's response speed and stability.
[0006] To achieve the above objectives, this application provides a downsampling method for big data visualization, comprising the following steps:
[0007] Initialize the chart settings and create a child thread that runs independently of the main thread;
[0008] The main thread divides the raw data into blocks, converts it to binary, and copies it zero to the child thread. The child thread generates view blocks based on the received data blocks and caches them.
[0009] The child thread merges multiple cached view blocks into view memory, performs downsampling on the view memory, and streams the downsampling results back to the main thread to complete the chart rendering.
[0010] When the rendering chart is triggered by a scaling operation, the main thread notifies the child thread of the data range targeted by the corresponding triggering event, so that the child thread performs a binary search in the view memory, and performs the downsampling process again based on the search result, and streams the downsampling result back to the main thread to update the chart display.
[0011] Optionally, the step of the main thread notifying the child thread of the corresponding data range for the triggering event when the rendered chart is triggered by a scaling operation includes:
[0012] When the rendering of the chart is triggered by a scaling operation, the main thread binds a timer to the trigger event corresponding to the scaling operation and monitors whether there is a new scaling operation before the timer reaches the preset duration.
[0013] If so, update the triggered event and reset the timer;
[0014] If not, when the preset duration is reached, the corresponding trigger event data range will be notified to the child thread.
[0015] Optionally, the sub-thread is created based on Web Worker technology, and the sub-thread and the main thread access relevant data based on worker memory.
[0016] Optionally, the downsampling method for big data visualization further includes:
[0017] When the child thread performs the downsampling process, it logically divides the data to be processed into blocks and calculates the step size so as to sample each block one by one.
[0018] Optionally, the downsampling method for big data visualization further includes:
[0019] When the child thread performs the downsampling process again based on the search results, it dynamically adjusts the sampling rate according to preset parameters.
[0020] The preset parameters include the scaling ratio corresponding to the trigger event, or the ratio of the data range targeted by the trigger event to the total data range in the view memory.
[0021] Optionally, the main thread divides the original data into blocks, converts it to binary, and zero-copys it to the child thread. The child thread then generates view blocks based on the received data blocks and caches them.
[0022] The main thread divides the raw data into multiple data blocks, allocates memory for each data block, performs binary conversion on each data block, and writes it into the corresponding memory.
[0023] The main thread transfers the memory ownership of each data block to the child thread;
[0024] The child thread reads each data block as a view block based on memory ownership and caches the view blocks as key-value pairs.
[0025] Optionally, the downsampling method for big data visualization further includes:
[0026] When the child thread performs the downsampling process, it allocates corresponding memory space and views according to the number of downsampling target points passed by the main thread to store the downsampling results.
[0027] Optionally, the downsampling method for big data visualization further includes:
[0028] When the child thread streams the downsampling results back to the main thread, the memory ownership of the downsampling results is transferred to the main thread.
[0029] To achieve the above objectives, this application also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the downsampling method for big data visualization described above.
[0030] To achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the downsampling method for big data visualization described above.
[0031] The big data visualization downsampling method, computer equipment, and computer-readable storage medium provided in this application achieve high-performance rendering and smooth interaction in real-time visualization scenarios with millions of data points through physical isolation between computation and rendering, zero-copy data storage and retrieval, and an efficient scaling and interaction mechanism, which greatly improves the system's response speed and stability.
[0032] Specifically, computationally intensive tasks are run independently on front-end sub-threads, decoupling data processing from front-end main thread rendering and ensuring smooth front-end page response; binary data is sampled and stored in blocks with a zero-copy mechanism, directly writing and manipulating large amounts of data into sub-thread memory to avoid redundant copying and frequent garbage collection; and an efficient scaling interaction algorithm is built to balance performance and data accuracy in high-frequency scaling scenarios, flexibly adapting to the needs of multiple industries and data-intensive applications. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of the steps of a big data visualization downsampling method in one embodiment of this application;
[0034] Figure 2 This is a schematic diagram of the operation process of the anti-shake mechanism triggered by rendering a chart in one embodiment of this application;
[0035] Figure 3 This is a schematic diagram of the internal architecture of a computer device according to an embodiment of this application.
[0036] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0037] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0038] Furthermore, descriptions using terms such as "first" and "second" in this application are for descriptive purposes only (e.g., to distinguish identical or similar features) and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include at least one of those features. Additionally, technical solutions from different embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If a combination of technical solutions is contradictory or impossible to implement, such a combination should be considered nonexistent and not within the scope of protection claimed in this application.
[0039] Reference Figure 1 In one embodiment, the downsampling method for big data visualization includes:
[0040] Step S10: Initialize the chart settings and create a child thread that runs independently of the main thread;
[0041] Step S20: The main thread divides the original data into blocks, converts it to binary, and zero-copys it to the child thread. The child thread generates view blocks based on the received data blocks and caches them.
[0042] Step S30: The child thread merges the cached multiple view blocks into view memory, performs downsampling on the view memory, and streams the downsampling results back to the main thread to complete the chart rendering.
[0043] Step S40: When the rendering chart is triggered by a scaling operation, the main thread notifies the child thread of the data range targeted by the corresponding triggering event, so that the child thread performs a binary search in the view memory, and performs the downsampling process again based on the search result, and streams the downsampling result back to the main thread to update the chart display.
[0044] In this embodiment, the execution terminal can be a computer device or other device or apparatus (such as a control device) that controls the computer device.
[0045] As described in step S10, to initialize the chart settings, you first need to install the Echarts library on the system, which can be done globally using the npm command.
[0046] After installation, globally reference the Echarts library in your project (the specific referencing method will vary depending on the technology stack used by your project). For example, include the Echarts library in the `<charts>` tag of your HTML file, and then in the `<head>` tag... <script>”标签中初始化一个默认的图表配置项option，并创建了一个Echarts实例"myChart”，最后使用setOption方法将配置项应用到图表上。
[0047] 在初始化图表的同时，通过创建worker文件来创建独立于主线程运行的子线程。其中，新建一个名为"myWorker.worker.js”的文件，该文件将定义数据的接收、处理与结果返回逻辑；在这个文件中，使用onmessage事件来监听主线程发送的数据，并使用postMessage方法将处理结果返回给主线程。
[0048] 在主线程中，通过new Worker()方法对myWorker.worker.js文件进行实例化，从而创建一个独立的子线程。
[0049] 这样，利用浏览器原生支持的Web Worker技术，构建了多线程并行计算与数据存储架构，可以将计算密集型任务移出主线程，并在独立线程中持续进行数据处理与缓存操作，避免了主线程因计算阻塞而引发的页面卡顿问题。在百万级数据的实时交互场景下，用户界面能够保持流畅响应，显著提升了业务分析的效率与使用体验。
[0050] 如步骤S20所述，在大数据可视化场景中，原始数据可能是一个庞大的数据集，为了更高效地处理和传输数据，主线程需要将原始数据进行分块、转换为二进制格式，并通过零拷贝的方式将数据传递给子线程。
[0051] 首先，主线程需要将原始数据按一定规则进行分块。分块的大小可以根据实际情况进行调整，例如根据数据的类型、数量以及系统的内存和性能等因素来确定；也可以是按设定的固定大小分块。
[0052] 可选的，所述子线程和所述主线程可以基于worker内存进行相关数据的存取。WebWorker为JavaScript提供了多线程能力，每个worker都有自己独立的执行环境和内存空间。worker内存是一个与主线程分离的内存区域，允许子线程在其中存储和操作数据，而不会影响主线程的执行，通过利用worker内存，主线程和子线程可以更高效地进行数据交互和共享。
[0053] 可选的，主线程可以使用ArrayBuffer、TypedArray等，将分块后的数据转换为二进制格式。其中，ArrayBuffer是一种用于表示通用的、固定长度的原始二进制数据缓冲区，而TypedArray则是基于ArrayBuffer的一种视图，用于以特定的数据类型(如Int8Array、Float32Array等)来操作ArrayBuffer中的数据。
[0054] 可选的，零拷贝是一种避免在数据传输过程中进行不必要的数据复制的技术。示例性的，如在JavaScript中，可以使用postMessage方法的第二个参数来实现零拷贝传递ArrayBuffer。当传递ArrayBuffer时，其所有权会从主线程转移到子线程，避免了数据的复制。
[0055] 子线程在接收到主线程传递的数据块后，需要根据这些数据生成视图块，并将视图块进行缓存。其中，在子线程的myWorker.worker.js文件中，可以使用onmessage事件来监听主线程发送的数据。
[0056] 生成视图块的具体逻辑取决于可视化的需求，例如可以根据数据块计算统计信息、绘制图形等。为了方便后续的合并和降采样处理，子线程需要将生成的视图块进行缓存，如使用相应的数组来存储视图块。
[0057] 这样，数据以二进制的形式存储在独立线程中，大幅降低主线程内存占用。
[0058] 如步骤S30所述，在子线程中，之前已经缓存了多个视图块。现在需要将这些视图块合并为一个统一的视图内存(即mergedView内存)，以便后续进行降采样处理。合并的具体方式取决于视图块的数据结构和可视化需求。视图内存是一种用于存储经过处理后的数据表示形式的内存区域，用于将多个视图块合并在一起。
[0059] 降采样是指在不损失太多信息的前提下，减少数据量以提高可视化的效率，可选的降采样方法包括简单的抽样、聚合等。
[0060] 为了提高性能，子线程可以将降采样结果流式返回给主线程，即分批次地将结果发送给主线程，而不是一次性发送。例如，子线程将降采样结果按批次截取，使用postMessage方法分批次发送给主线程。
[0061] 在主线程中，监听子线程返回的数据，逐步更新Echarts图表，实现流式渲染。例如，主线程通过onmessage事件监听子线程发送的数据，将新数据添加到图表的系列数据中，并更新x轴数据，然后使用setOption方法更新图表显示。
[0062] 如步骤S40所述，在主线程中，需要监听图表的缩放事件，当缩放操作触发时，确定触发事件所涉及的数据区间，并将该区间信息发送给子线程。即当用户对渲染的图表进行缩放操作时，主线程会捕捉到该操作，并将缩放操作所涉及的数据区间信息传递给子线程。
[0063] 子线程接收到主线程发送的事件通知后，会在已有的视图内存中使用二分查找算法定位到相应的数据范围，然后基于这个查找结果再次进行降采样处理，最后将新的降采样结果流式返回给主线程。
[0064] 主线程监听子线程返回的数据，将新的降采样结果更新到图表中。
[0065] 这样，可以实现高效的缩放交互处理，在高频缩放场景下兼顾性能与数据精度，可以灵活适配多行业、多数据密集型应用需求。
[0066] 在一实施例中，通过计算与渲染的物理隔离、零拷贝数据存储与读取以及高效的缩放交互机制，实现了在百万级数据实时可视化场景下的高性能渲染与流畅交互，大幅提升了系统的响应速度与稳定性。
[0067] 其中，将计算密集型任务独立运行于前端子线程，实现数据处理与前端主线程的渲染解耦，确保前端页面响应流畅；采样二进制数据分块存储与零拷贝机制，将大量数据直接写入并操作与子线程内存，避免冗余复制与频繁垃圾回收；构建高效的缩放交互算法，在高频缩放场景下兼顾性能与数据精度，灵活适配多行业、多数据密集型应用需求。
[0068] 例如在一些大数据可视化平台实施本方案时，构建的高效降采样大数据可视化体系实现了对百万级数据的高性能渲染与交互，用户通过选择降采样方法和目标点数即可完成降采样数据以及缩放操作的图表展示。这样在保证可视化流畅性的同时，显著提升了平台在大规模数据处理下的响应速度与稳定性。
[0069] 在一实施例中，在上述实施例基础上，所述当渲染图表被缩放操作触发时，主线程向子线程通知相应的触发事件针对的数据区间的步骤包括：
[0070] 当渲染图表被缩放操作触发时，主线程为缩放操作对应的触发事件绑定计时器，并监测在计时达到预设时长前是否有新的缩放操作；
[0071] 若是，则更新触发事件并重新计时；
[0072] 若否，则在计时达到预设时长时，向子线程通知相应的触发事件针对的数据区间。
[0073] 本实施例中，为图表的缩放事件绑定监听器，这样每当图表发生缩放操作时，系统就会感知到该事件并做出相应处理。
[0074] 可选的，参照图2，当缩放操作触发时，主线程会为该缩放操作对应的触发事件绑定一个计时器，防抖时间对应预设时长可以根据实际情况需要设置(如设置的防抖时长为300ms)，其目的是防止在短时间内频繁触发缩放操作而导致大量的计算任务堆积。例如，用户可能在短时间内连续多次缩放图表，如果每次缩放都立即触发后续的计算和请求，会给系统带来较大的负担，甚至可能导致界面卡顿。
[0075] 每次新的缩放操作触发时，会先清理上一次还未执行的防抖定时器。这确保了在预设时长内，如果有新的缩放操作发生，之前未执行的定时器会被取消，只有最后一次缩放操作在预设时长后才会真正触发后续的请求，从而保证缩放操作只触发一次请求，避免了不必要的频繁计算。
[0076] 在设置的防抖计时过程中，如果监测到有新的缩放操作发生，系统会更新触发事件，并重新开始计时。这意味着只有当用户在预设时长内没有再进行缩放操作时，才会继续后续步骤。
[0077] 若在计时达到预设时长时都没有新的缩放操作，主线程会向子线程通知相应的触发事件针对的数据区间。具体来说，主线程会从缩放事件的参数中获取当前x轴的范围xRange，这个xRange就代表了触发事件针对的数据区间，然后将其传递给子线程(worker)。
[0078] 子线程接收到主线程传递的xRange后，会在之前缓存的视图内存(即mergedView内存)中进行二分查找。由于mergedView内存中存储了之前生成并合并的视图数据，二分查找可以利用其有序性，快速定位到与xRange相关的数据，从而实现快速的数据过滤。例如，如果之前缓存的数据是按照x轴的值有序排列的，通过二分查找可以高效地找到落在当前x轴范围xRange内的数据。
[0079] 对过滤后的数据再次执行相应的降采样处理，并将处理结果流式返回给主线程。这样主线程可以根据返回的降采样结果更新图表显示，实现流畅的缩放体验。通过这种方式，即使在处理大数据量的可视化图表时，也能保证在缩放操作时图表的更新能够快速响应，给用户带来良好的交互体验。
[0080] 在一实施例中，在图表缩放交互中引入防抖机制，避免在短时间内重复触发大量计算任务，同时结合worker的二分查找算法快速定位缩放范围内的数据，实现缩放的降采样与渲染。实测显示该算法显著加快了缩放响应速度，从而在频繁缩放的场景下，依然能够提供流畅、稳定的用户体验。
[0081] 在一实施例中，在上述实施例的基础上，所述大数据可视化的降采样方法，还包括：
[0082] 子线程在执行所述降采样处理时，对待处理数据进行逻辑分块并计算步长，以对分块进行逐个采样。
[0083] 本实施例中，可以根据系统默认设置，确定降采样所使用的方法。如默认采用随机降采样方法，但也可以根据实际需求选择其他合适的方法。
[0084] 根据预先设定的规则或者根据数据的特性，将待处理数据进行逻辑上的分块。逻辑分块的目的是将大规模的数据划分为相对较小的、便于处理的子数据集合，以便后续进行局部的采样操作。
[0085] 例如，假设待处理数据是一个包含时间序列的数据集，数据以时间顺序排列。可以按照固定的时间间隔(例如每小时、每天等)对数据进行分块；或者根据数据的数量，将数据均匀地划分为若干个大小相同的块。
[0086] 步长是指在每个分块内进行采样时，数据点之间的间隔距离。通过计算步长，可以确定在每个分块内需要采样的数据点的位置。步长的计算与分块的大小和降采样的目标有关。例如，如果要将每个分块的数据量降采样为原来的一半，那么步长可以设置为2，表示每隔一个数据点进行一次采样。具体计算公式可以根据实际的降采样需求进行调整。假设一个分块包含100个数据点，需要将其降采样为50个数据点，那么步长可以计算为100 / 50＝2。
[0087] 对于每个分块，固定选取该分块中的第一个x值作为采样结果的一部分。这里的x值可以代表数据的某个维度，例如时间、横坐标等。
[0088] 根据选定的降采样方法，计算每个分块中与第一个x值对应的y值。如果采用随机降采样方法，可能会在分块内随机选择一个y值作为采样结果；如果采用其他方法，例如平均值法，则会计算分块内所有y值的平均值作为采样结果。
[0089] 在采样过程中，需要做好越界保护，确保最后一个分块也能被正确处理。由于最后一个分块的大小可能与其他分块不同，可能会出现数据不足的情况。因此，在计算步长和采样时，需要对最后一个分块进行特殊处理，避免出现数组越界等错误。
[0090] 假设最后一个分块只包含20个数据点，而步长为2。在采样时，需要确保不会超出这20个数据点的范围。可以通过判断当前采样位置是否超出分块的大小来进行越界保护。
[0091] 将每个分块的采样结果进行汇总，形成降采样后的数据。这些数据将被流式返回至主线程，用于完成图表的渲染或者更新图表的显示。
[0092] 这样，子线程可以对待处理数据进行逻辑分块并计算步长，实现对分块的逐个采样，从而完成降采样处理，提高数据的处理效率和图表的渲染性能。
[0093] 在一实施例中，在上述实施例的基础上，所述大数据可视化的降采样方法，还包括：
[0094] 子线程基于查找结果再次执行所述降采样处理时，根据预设参数动态调整采样率；
[0095] 其中，预设参数包括触发事件对应的缩放比例，或触发事件针对的数据区间占视图内存的总数据区间的比值。
[0096] 本实施例中，在原有的大数据可视化降采样方法基础上，当子线程基于查找结果再次执行降采样处理时，会根据预设参数动态调整采样率。这里的预设参数主要包括触发事件对应的缩放比例，或者触发事件针对的数据区间占视图内存的总数据区间的比值。通过动态调整采样率，可以更灵活地适应不同的可视化需求，在保证可视化效果的同时，提高处理效率。
[0097] 可选的，当用户对渲染图表进行缩放操作时，会产生一个缩放比例。例如，用户将图表放大到原来的2倍，缩放比例就是2；缩小到原来的0.5倍，缩放比例就是0.5。这个缩放比例反映了用户对图表细节的需求程度，放大操作通常意味着用户希望看到更多的细节，需要更高的采样率；缩小操作则可能不需要那么多细节，可以降低采样率。
[0098] 可选的，视图内存中存储了经过合并的多个视图块所包含的总数据区间。当触发事件发生时，会指定一个针对的数据区间。计算该数据区间占总数据区间的比值，可以帮助判断当前需要处理的数据范围大小。例如，如果比值较小，说明只需要处理视图内存中一小部分数据，可能需要提高采样率以保证这部分数据的可视化效果；如果比值较大，则可以适当降低采样率。
[0099] 可选的，当缩放比例大于1时，意味着图表被放大，用户需要查看更多细节。子线程可以根据缩放比例的大小相应地提高采样率。例如，设置一个采样率调整系数，假设缩放比例为2，调整系数为0.5，那么新的采样率可以是原来采样率乘以(1+缩放比例*调整系数)。即如果原来的采样率是10％，新的采样率可以调整为10％*(1+2*0.5)＝20％。
[0100] 可选的，当缩放比例小于1时，图表被缩小，不需要过多的细节。子线程可以降低采样率。同样使用调整系数，假设缩放比例为0.5，新的采样率可以是原来采样率乘以(1-(1-缩放比例)*调整系数)。即原来采样率为10％，新的采样率可以调整为10％*(1-(1-0.5)*0.5)＝7.5％。
[0101] 可选的，如果触发事件针对的数据区间占视图内存总数据区间的比值小于某个阈值(例如0.2)，说明只需要处理一小部分数据，为了保证这部分数据的可视化质量，子线程可以提高采样率。可以设置一个固定的提高幅度，如将采样率提高10％。
[0102] 可选的，如果比值大于某个阈值(例如0.8)，说明需要处理的数据范围较大，可以适当降低采样率。例如，将采样率降低10％。
[0103] 子线程根据调整后的采样率对查找结果再次执行降采样处理。在处理过程中，子线程会将降采样结果流式返回至主线程，主线程利用这些结果更新图表显示，让用户能够实时看到调整后的可视化效果。
[0104] 在一实施例中，根据不同的缩放操作和数据区间动态调整采样率，可以在用户需要更多细节时提供更高质量的可视化，满足用户对数据细节的查看需求。在不需要过多细节的情况下降低采样率，减少了数据处理量，提高了处理效率，避免不必要的计算资源浪费。
[0105] 在一实施例中，在上述实施例的基础上，所述主线程将原始数据分块、转为二进制并零拷贝到子线程，子线程根据接收到数据块生成视图块并缓存的步骤包括：
[0106] 主线程将原始数据切分成多个数据块，且为每个数据块分配内存，并将每个数据块进行二进制转换后，写入相应的内存中；
[0107] 主线程向子线程转移各数据块相应的内存所有权；
[0108] 子线程基于内存所有权将各数据块读取为视图块，并以键值对的方式缓存视图块。
[0109] 本实施例中，在大数据可视化场景中，主线程通常会接收到大量的原始数据。为了便于处理和传输，主线程可以将这些原始数据按照固定的大小切分成多个数据块。固定大小的设定可以根据实际情况进行调整，例如根据内存使用情况、数据传输的效率等因素。这样做的好处是可以将大规模的数据分解为较小的、易于管理的部分，提高后续处理的效率。
[0110] 在将原始数据分块之后，主线程需要为每个数据块分配一段连续的内存空间。这里使用ArrayBuffer来实现，ArrayBuffer是一种用于表示通用的、固定长度的二进制数据缓冲区的对象。每个ArrayBuffer可以存储一定数量的字节，其大小在创建时就已经确定。
[0111] 可选的，对于每个数据块，主线程会使用Float64Array视图将时间或者数值型的字段统一转换为二进制格式，然后直接写入相应的ArrayBuffer内存中。Float64Array是一种类型化数组，可以以64位浮点数的形式访问ArrayBuffer中的数据。
[0112] 在完成数据块的二进制转换和写入内存之后，主线程需要将各数据块相应的内存所有权转移给子线程(即worker)。这里使用postMessage方法发送数据块给worker，同时通过Transferable接口来转移内存所有权，避免数据的拷贝。这样可以大大提高数据传输的效率，减少内存的使用。
[0113] 子线程在onmessage事件中接收到主线程传递过来的内存后，会直接使用传递过来的ArrayBuffer创建Float64Array视图来读取数据。由于内存所有权已经转移，子线程可以直接访问这些内存，无需进行数据的复制。
[0114] 可选的，子线程会将读取到的视图块以键值对的方式进行缓存，方便后续的使用。键可以是数据块的编号或者其他唯一标识，值就是对应的视图块。这样在需要使用这些数据时，可以快速地通过键来查找和获取相应的视图块。
[0115] 通过以上步骤，主线程成功地将原始数据分块、转为二进制并零拷贝到子线程，子线程也根据接收到的数据块生成了视图块并进行了缓存，为后续的降采样处理和图表渲染做好了准备。
[0116] 在一实施例中，在上述实施例的基础上，所述大数据可视化的降采样方法，还包括：
[0117] 子线程在执行所述降采样处理时，根据主线程传递的降采样目标点数分配相应的内存空间和视图，用于存放降采样结果。
[0118] 本实施例中，子线程首先会将之前缓存的多个Float64Array视图块合并成一段连续的内存mergedView，形成视图内存，以便后续进行降采样处理。
[0119] 可选的，子线程可以通过遍历缓存的视图块，将它们的数据依次复制到一个新的ArrayBuffer中，然后创建一个新的Float64Array视图来引用这个ArrayBuffer，从而得到mergedView，作为视图内存。
[0120] 可选的，主线程在需要进行降采样处理时，会向子线程传递降采样目标点数。这个目标点数表示经过降采样后的数据点数量，是根据图表的显示需求、用户的缩放操作等因素确定的。子线程接收到主线程传递的降采样目标点数后，会根据这个点数分配相应的内存空间和视图，用于存放降采样结果。
[0121] 子线程分配好内存空间和视图后，会基于视图内存进行降采样处理，并将降采样结果存入之前分配的视图中。子线程完成降采样处理后，会将降采样结果以流式的方式返回给主线程，主线程可以使用这些结果完成图表的渲染或更新。
[0122] 在一实施例中，子线程在执行降采样处理时，能够根据主线程传递的降采样目标点数准确地分配内存空间和视图，存放降采样结果，并将结果返回给主线程，从而实现大数据可视化的降采样处理和图表渲染。
[0123] 在一实施例中，在上述实施例的基础上，所述大数据可视化的降采样方法，还包括：
[0124] 子线程流式返回降采样结果至主线程时，向主线程转移降采样结果相应的内存所有权。
[0125] 本实施例中，子线程在流式返回降采样结果给主线程的时候，可以把降采样结果对应的内存所有权转移给主线程。如在JavaScript里，可以借助postMessage方法搭配Transferable接口达成这一目的。当使用Transferable接口时，数据的所有权会直接从子线程转移到主线程，而不会进行数据拷贝，这样能显著提升数据传输效率，减少内存占用。
[0126] 主线程在接收到子线程传递过来的降采样结果后，就能够直接使用这个ArrayBuffer创建视图，进而用于图表的渲染或者更新。
[0127] 在一实施例中，通过转移内存所有权避免数据拷贝，能够减少数据传输过程中的时间开销，从而加快降采样结果的返回速度，提升整个大数据可视化系统的响应性能。由于不进行数据拷贝，系统无需额外的内存来存储拷贝的数据，降低了内存的使用量，尤其在处理大规模数据时，这种优势更为明显。
[0128] 此外，本申请实施例中还提供一种计算机设备，该计算机设备内部架构可以如图3所示，包括通过系统总线连接的处理器、存储器、通信接口和输入接口。其中，该处理器用于提供计算和控制能力。该存储器包括非易失性存储介质和内存储器。该非易失性存储介质存储有操作系统、计算机程序和数据库。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该数据库用于存储计算机程序调用的数据。该通信接口用于与外部的终端进行数据通信。该输入接口用于接收外部设备输入的信号。该计算机程序被处理器执行时以实现一种如以上实施例所述的大数据可视化的降采样方法。
[0129] 本领域技术人员可以理解，图3中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定。
[0130] 此外，本申请还提出一种计算机可读存储介质，所述计算机可读存储介质包括计算机程序，所述计算机程序被处理器执行时实现如以上实施例所述的大数据可视化的降采样方法的步骤。可以理解的是，本实施例中的计算机可读存储介质可以是易失性可读存储介质，也可以为非易失性可读存储介质。
[0131] 综上所述，为本申请实施例中提供的大数据可视化的降采样方法、计算机设备和计算机可读存储介质，通过计算与渲染的物理隔离、零拷贝数据存储与读取以及高效的缩放交互机制，实现了在百万级数据实时可视化场景下的高性能渲染与流畅交互，大幅提升了系统的响应速度与稳定性。
[0132] 本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的和实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和 / 或易失性存储器。非易失性存储器可以包括只读存储器(Read-OnlyMemory，ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(Random Access Memory，RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM通过多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双速据率SDRAM(SSRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
[0133] 需要说明的是，在本文中，术语"包括”、"包含”或者其任何其它变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、装置、物品或者方法不仅包括那些要素，而且还包括没有明确列出的其它要素，或者是还包括为这种过程、装置、物品或者方法所固有的要素。在没有更多限制的情况下，由语句"包括一个……”限定的要素，并不排除在包括该要素的过程、装置、物品或者方法中还存在另外的相同要素。
[0134] 以上所述仅为本申请的优选实施例，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其它相关的技术领域，均同理包括在本申请的专利保护范围内。< / script>
Claims
1. A downsampling method for big data visualization, characterized in that, include: Initialize the chart settings and create a child thread that runs independently of the main thread; The main thread divides the raw data into blocks, converts it to binary, and zero-copys it to the child thread. The child thread generates view blocks based on the received data blocks and caches them. The child thread merges multiple cached view blocks into view memory, performs downsampling on the view memory, and streams the downsampling results back to the main thread to complete the chart rendering. When the rendering chart is triggered by a scaling operation, the main thread notifies the child thread of the data range corresponding to the triggering event, so that the child thread performs a binary search in the view memory, and performs the downsampling process again based on the search result, and streams the downsampling result back to the main thread to update the chart display.
2. The downsampling method for big data visualization as described in claim 1, characterized in that, The step of the main thread notifying the child thread of the corresponding data range for the triggering event when the rendered chart is triggered by a scaling operation includes: When the rendering of the chart is triggered by a scaling operation, the main thread binds a timer to the trigger event corresponding to the scaling operation and monitors whether there is a new scaling operation before the timer reaches the preset duration. If so, update the triggered event and reset the timer; If not, when the preset duration is reached, the corresponding trigger event data range will be notified to the child thread.
3. The downsampling method for big data visualization as described in claim 1, characterized in that, The sub-thread is created based on WebWorker technology, and the sub-thread and the main thread access relevant data based on worker memory.
4. The downsampling method for big data visualization as described in any one of claims 1-3, characterized in that, The downsampling method for big data visualization also includes: When the child thread performs the downsampling process, it logically divides the data to be processed into blocks and calculates the step size so as to sample each block one by one.
5. The downsampling method for big data visualization as described in any one of claims 1-3, characterized in that, The downsampling method for big data visualization also includes: When the child thread performs the downsampling process again based on the search results, it dynamically adjusts the sampling rate according to preset parameters. The preset parameters include the scaling ratio corresponding to the trigger event, or the ratio of the data range targeted by the trigger event to the total data range in the view memory.
6. The downsampling method for big data visualization as described in any one of claims 1-3, characterized in that, The main thread divides the raw data into blocks, converts it to binary, and zero-copys it to the child thread. The child thread then generates and caches view blocks based on the received data blocks. The main thread divides the raw data into multiple data blocks, allocates memory for each data block, performs binary conversion on each data block, and writes it into the corresponding memory. The main thread transfers the memory ownership of each data block to the child thread; The child thread reads each data block as a view block based on memory ownership and caches the view blocks as key-value pairs.
7. The downsampling method for big data visualization as described in claim 1, characterized in that, The downsampling method for big data visualization also includes: When the child thread performs the downsampling process, it allocates corresponding memory space and views according to the number of downsampling target points passed by the main thread to store the downsampling results.
8. The downsampling method for big data visualization as described in claim 6, characterized in that, The downsampling method for big data visualization also includes: When the child thread streams the downsampling results back to the main thread, the memory ownership of the downsampling results is transferred to the main thread.
9. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the downsampling method for big data visualization as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the downsampling method for big data visualization as described in any one of claims 1 to 8.