Data visualization method and system, storage medium, equipment and program product

Abstract

The embodiment of the invention provides a data visualization method and system, a storage medium, equipment and a program product. The method comprises the steps of determining image data and motion control data for visualization of a current playing frame based on the current playing frame in a playing controller; performing an image rendering operation using the determined image data, and displaying an image rendering result in a first display area; and performing a motion state rendering operation using the determined motion control data, and displaying a motion state rendering result in the second display area. According to the method, an image rendering result and a motion state rendering result are synchronously displayed in different areas of a visual interface respectively by utilizing a playing controller and a current playing frame serving as a time pointer, so that the image rendering result and the motion state rendering result can be played according to the same time axis; the image rendering result and the motion state rendering result are integrated in different areas of the same interface, and a user can master the image rendering result and the motion state rendering result without switching views.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of robotics technology, and in particular to a data visualization method and system, a storage medium, a device, and a program product. Background Art

[0002] In the field of embodied intelligence, multimodal data (such as image streams, joint angles, and end-user poses) collected by robots through multimodal sensors (such as joint encoders and force sensors) is a key basis for training and optimizing intelligent models. This data typically requires manual review and annotation to ensure its quality and validity, which in turn affects the accuracy and robustness of subsequent model training. However, existing data visualization solutions face numerous technical bottlenecks.

[0003] Based on this, embodiments of the present application provide data visualization methods and systems, storage media, devices, and program products to improve related technologies. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a data visualization method and system, storage medium, device and program product to synchronously display relevant multimodal data of a robot.

[0005] The purpose of the embodiments of the present application is achieved by using the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a data visualization method, which is applied to a data visualization system, wherein the visualization interface in the system includes a playback controller, a first display area, and a second display area. The method includes: based on the current playback frame in the playback controller, determining image data corresponding to the robot for visualization of the current playback frame and motion control data related to the robot state; using the determined image data to perform an image rendering operation, and displaying the image rendering result in the first display area; using the determined motion control data to perform a motion state rendering operation, and displaying the motion state rendering result in the second display area.

[0007] In some embodiments, the determined image data includes image data of multiple channels, and the process of performing an image rendering operation using the determined image data includes: in response to a selection operation on one of the channels, calling the image data of the corresponding channel for image rendering.

[0008] In some embodiments, the determined motion control data includes joint angle data from a first specified frame to a currently played frame, and the process of performing a motion state rendering operation using the determined motion control data includes: generating line chart data based on the joint angle data from the first specified frame to the currently played frame, and drawing a line chart using the line chart data.

[0009] In some embodiments, the determined motion control data includes joint angle data of the current playback frame, the visualization interface also includes a third display area, and the method also includes: based on the joint angle data of the current playback frame, driving the robot model of the robot to perform posture rendering, and displaying the model rendering result in the third display area.

[0010] In some embodiments, the process of determining image data for visualization of the current playback frame includes: downloading an image data packet that has undergone multiple compression processes from an object storage system; decompressing the downloaded image data packet and storing it in a local database; and reading image data corresponding to the current playback frame from the local database.

[0011] In some embodiments, the image data packet is organized by time period, and the compression process of the image data packet includes: performing file-level merging on multiple image frames of the robot within the corresponding time period to achieve primary compression; and performing secondary compression on the file obtained by the primary compression to obtain the image data packet.

[0012] In some embodiments, the visualization interface also includes a label alignment area, and the method also includes: displaying at least one first identifier in the label alignment area, the at least one first identifier including an action step identifier and / or a key frame identifier; in response to a selection operation on the first identifier, performing an operation of aligning the first identifier to the current playback frame, and associating the first identifier with the frame index information of the current playback frame and saving it; wherein, the first identifier is a graphic mark or a text label, and the selection operation includes a drag operation and / or a double-click operation.

[0013] In some embodiments, the method further includes: displaying a second identifier in the playback controller, the second identifier being used to indicate the position of the current playback frame; in response to a drag operation on the second identifier, updating the current playback frame to a corresponding playback frame at the dragged position of the second identifier; wherein the second identifier is a marking line or a graphic mark.

[0014] In some embodiments, the method also includes: based on the set time period corresponding to the action step, using a third identifier to perform a marking operation on the time axis of the playback controller; the set time period is determined according to the preset task action information; wherein, the third identifier is a line segment or bar mark located below the time axis, or a line or a floating block graphic mark covering the time axis for marking the start and end time range.

[0015] In second aspect, an embodiment of the present application provides a data visualization system, which includes: a visualization interface, including a playback controller, a first display area, and a second display area; a data determination module, which determines, based on the current playback frame in the playback controller, image data corresponding to the robot for visualization of the current playback frame and motion control data related to the robot state; an image processing module, which uses the determined image data to perform an image rendering operation and display the image rendering result in the first display area; and a state processing module, which uses the determined motion control data to perform a motion state rendering operation and display the motion state rendering result in the second display area.

[0016] In some embodiments, the determined motion control data includes joint angle data of the current playback frame, the visualization interface also includes a third display area, and the system also includes: a model processing module, which is used to drive the robot model of the robot to perform posture rendering based on the joint angle data of the current playback frame, and display the model rendering result in the third display area.

[0017] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above methods when executing the computer program.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements any of the above methods.

[0020] The embodiments of the present application provide a data visualization method and system, a storage medium, a device and a program product. By utilizing a unified playback controller and a current playback frame as a "time pointer", the image rendering results obtained from the image data and the motion state rendering results obtained from the motion control data can be synchronously displayed in the first display area and the second display area of ​​the visualization interface, respectively, and can then be played according to the same timeline; the image rendering results obtained from the image data and the motion state rendering results obtained from the motion control data are respectively integrated in the first display area and the second display area of ​​the same visualization interface, and the user can grasp the image rendering results and motion state rendering results of the robot without switching views. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The embodiments of the present application are further described below with reference to the accompanying drawings and specific implementation methods.

[0022] Figure 1a It is a flowchart of a data visualization method provided in an embodiment of the present application.

[0023] Figure 1b This is a schematic diagram of a visualization interface provided in an embodiment of the present application.

[0024] Figure 2 This is a flowchart of another data visualization method provided in an embodiment of the present application.

[0025] Figure 3 This is a module diagram of a data visualization system provided in an embodiment of the present application.

[0026] Figure 4 This is a schematic diagram of another visual interface provided in an embodiment of the present application.

[0027] Figure 5 This is a module diagram of another data visualization system provided in an embodiment of the present application.

[0028] Figure 6 This is a schematic diagram of another visual interface provided in an embodiment of the present application.

[0029] Figure 7 This is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of the embodiments of the present application.

[0031] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0032] In the field of embodied intelligence, the data collected by robots through multimodal sensors (such as cameras and joint encoders) can include one or more of these multimodal data types: RGB-D images, joint angle data, and end-effector pose. RGB-D images are unified data formed by aligning and fusing pixel information from the red (R), green (G), and blue (B) channels with depth information.

[0033] Through in-depth analysis of relevant data visualization methods, the applicant found that they may have the following defects: 1) Data fragmentation problem: Since image data (such as RGB-D images) and motion control data (such as joint angle data) are usually displayed in different software modules or interfaces, reviewers need to repeatedly switch views to compare the robot status at the same time, which is extremely inefficient; 2) Time asynchrony problem: Since the playback of image data and motion control data is usually controlled by different independent modules, it is difficult to ensure strict synchronization; 3) Poor interactivity problem: Due to the dispersion of data, it is difficult for users to simultaneously adjust the playback progress of all image data and motion control data through unified control (such as a progress bar), resulting in cumbersome operation.

[0034] Therefore, in order to improve the defects of the above-mentioned related methods, an embodiment of the present application provides a data visualization method. This method uses the current playback frame as a "time pointer" and a unified playback controller to synchronize the image rendering results obtained from the image data and the motion state rendering results obtained from the motion control data in the first display area and the second display area of ​​the visualization interface, respectively, so that they can be played strictly according to the same timeline; by adopting an integrated visualization interface, the image rendering results obtained from the image data and the motion state rendering results obtained from the motion control data are integrated in different areas of the same interface (such as the first and second display areas), and the user can fully understand the robot status without switching views.

[0035] See also Figure 1a , Figure 1a It is a flowchart of a data visualization method provided in an embodiment of the present application.

[0036] To address the issues of multimodal data fragmentation, time asynchrony, and poor interactivity in related data visualization methods, an embodiment of the present application provides a data visualization method. This method can be applied to a data visualization system, wherein the visualization interface in the system includes a playback controller, a first display area, and a second display area. The method includes steps S100 to S300.

[0037] Step S100: Based on the current playing frame in the playing controller, determine image data corresponding to the robot for visualization of the current playing frame and motion control data related to the robot state.

[0038] Step S200: performing an image rendering operation using the determined image data, and displaying the image rendering result in the first display area.

[0039] Step S300: performing a motion state rendering operation using the determined motion control data, and displaying the motion state rendering result in the second display area.

[0040] It is understandable that the data visualization system may refer to the hardware and software architecture for realizing the visualization of multimodal data. For example, the data visualization system may include a visualization interface. The data visualization system may also include a data processing module, a storage system, etc. The data visualization system may be used to integrate one or more of the multimodal data collected by the robot. The visualization interface may refer to a graphical carrier for user interaction. The visualization interface may include a playback controller, a first display area, and a second display area. The playback controller may be used to uniformly drive the frame-level synchronization of multimodal data. The playback controller is a component for users to interact with audio and video content, etc. The playback controller allows users to easily control the playback process and enhance the experience through intuitive buttons, sliders, and prompt information.

[0041] In some embodiments, in step S100, the current playback frame in the playback controller may refer to a data frame corresponding to a specific time point selected by the user on a visual interface (such as a progress bar or a timeline) or currently being rendered by the playback controller.

[0042] It is understandable that the "current playback frame in the playback controller" can be used to uniformly control the synchronous rendering of multimodal data; its essence is an index mark pointing to the time series data currently being processed or displayed, which can be understood as the "time pointer" of the entire data visualization system. The current playback frame can serve as a unified timing benchmark for the synchronous rendering of multimodal data. For example, the current playback frame can be identified by a timestamp or frame sequence number. The image data and motion control data can be strictly aligned through the above-mentioned identification. With the real-time update of user operations or playback progress, the loading and rendering of subsequent data can be triggered based on the current playback frame. For example, when the playback controller plays to the 50th frame, the relevant modules automatically obtain the image data and motion control data of the 50th frame.

[0043] In some embodiments, in step S100, image data (e.g., RGB-D image) used to visualize the current playback frame and motion control data related to the robot state (e.g., joint angle data) can be synchronously queried from relevant modules (e.g., storage systems) based on the current playback frame. The corresponding image data and motion control data can then be obtained.

[0044] In some embodiments, in step S100, the image data corresponding to the robot used to visualize the current playback frame strictly matches the time point of the current playback frame. This image data can be raw or pre-processed image data acquired by the robot's multimodal sensors. This image data can be used to render in the first display area of ​​the visualization interface to reflect the robot's environmental perception state at that moment. This image data can be derived from a time-series image stream acquired by a multi-channel camera.

[0045] In some embodiments, in step S100, the motion control data related to the robot state can strictly match the time point of the current playback frame. The above-mentioned motion control data may include time-series motion parameters for describing the physical action state of the robot. The above-mentioned time-series motion parameters may include joint angle data. The above-mentioned time-series motion parameters may also include end effector posture, etc. Those skilled in the art can determine the specific category of motion control data according to the actual scene requirements, and there is no limitation here. The motion control data can reflect the robot's action state. The above-mentioned motion control data can be extracted from the same data source through the index of the current playback frame, etc.

[0046] In some embodiments, the motion control data in step S100 may only include the motion control data corresponding to the current playback frame. The motion control data in step S100 may also include the motion control data corresponding to the first playback frame (such as the 1st frame) to the current playback frame. The motion control data in step S100 may also include the motion control data from the iath (a is less than i) frame to the i-th frame (the current playback frame). Similarly, the image data in step S100 may only include the image data corresponding to the current playback frame. The image data in step S100 may also include the image data corresponding to the first playback frame (such as the 1st frame) to the current playback frame. The motion control data in step S100 may also include the image data from the iath (a is less than i) frame to the i-th frame (the current playback frame).

[0047] In some embodiments, the playback controller can provide functions such as a progress bar, a play / pause button, and a frame jump. User interaction (such as dragging a progress bar) can trigger the update of the current playback frame. The index of the current playback frame (such as a timestamp or frame number) can be used as a core parameter. The index of the current playback frame can be used to synchronize the rendering of multimodal data. This method can use the current playback frame as a "time pointer" and a unified playback controller so that the image rendering result obtained from the image data and the motion state rendering result obtained from the motion control data are displayed synchronously in the first display area and the second display area of ​​the visualization interface respectively, and can then be played strictly according to the same timeline; and the user only needs to operate the progress bar in the playback controller to simultaneously view the robot's environmental perception state and motion state at the same time without manual alignment. When the user adjusts the playback progress (such as jumping to a certain frame), the system can automatically and synchronously update the content of all display areas to ensure data consistency in all display areas. In other words, the user experience can be optimized through a unified control method (such as a progress bar, frame jump).

[0048] In some embodiments, in step S200, the purpose of "performing an image rendering operation using the determined image data" may include: converting pixel data (such as the determined image data) into a visual image frame. For example, the underlying graphics interface may be called to perform an image rendering operation. The image rendering operation may involve image enhancement (such as denoising, contrast adjustment), etc., to improve the clarity of the image rendering result. The specific process of the image rendering operation can directly adopt relevant technologies, and no specific restrictions are made here. Afterwards, the image rendering result (such as the image obtained by the image rendering operation) can be displayed in the first display area. For example, the image rendering result displayed in the first display area can be refreshed in real time according to the frame changes of the playback controller (such as play, pause, frame skip).

[0049] It is understood that the purpose of "displaying the image rendering result in the first display area" in step S200 is to provide an intuitive display of the robot's perception perspective, helping the user determine whether environmental objects are correctly identified (such as the location of the grasped target) and whether there are any anomalies in the image data (such as blur or occlusion). For example, by displaying the real-time grasping image from the robot's left-hand camera, it can assist in verifying the rationality of the grasping action.

[0050] In some embodiments, in step S300, the purpose of "performing a motion state rendering operation using the determined motion control data" may include converting raw data (such as motion control data) into a visual image frame. The specific process of the image rendering operation can directly adopt relevant technologies and is not specifically limited here. For example, the motion state rendering operation can be performed using ECharts (an open source, web-based, cross-platform framework that supports the rapid creation of interactive visualizations) to draw the motion state related to the robot. Afterwards, the motion state rendering result can be displayed in the second display area.

[0051] In some embodiments, the image rendering operation in step S200 and the motion state rendering operation in step S300 can be independent of each other. The image rendering result and the motion state rendering result can be visually synchronized by sharing the frame index of the current playback frame, thereby minimizing performance degradation caused by code coupling.

[0052] It is understandable that the order of step S200 and step S300 can be interchanged. Step S200 and step S300 can also be executed simultaneously.

[0053] As can be seen, in some embodiments, an integrated visualization interface can be used to integrate the image rendering results obtained from image data and the motion state rendering results obtained from motion control data into different areas of the same interface (such as the first and second display areas), allowing users to fully understand the robot status without switching views. In other words, users do not need to switch between different modules, which can reduce the number of operation steps. Strict alignment of all relevant data can reduce the risk of misjudgment.

[0054] During the operation of an embodied intelligent robot, multiple cameras are usually installed, such as a head camera that provides a global view, a left-hand camera that provides an end-effector view, and a right-hand camera that provides a collaborative operation view. However, the images from different cameras usually require switching between different software or tabs to view, which makes the operation cumbersome, and it is impossible to quickly compare images from different perspectives at the same time, resulting in inefficient data review. In addition, if the video streams from the head, left-hand, and right-hand cameras are loaded simultaneously, it will cause an HTTP request storm, causing queuing delays, and unused channel data will still occupy memory.

[0055] In order to facilitate users to view multi-view image data, in some embodiments, the determined image data may include image data of multiple channels. Figure 1b , Figure 1bis a schematic diagram of a visualization interface provided by an embodiment of the present application. There may be multiple first display areas 102. In step S200, performing an image rendering operation using the determined image data may include: performing image rendering using the image data of each channel for some or all channels, and displaying the image in one of the first display areas 102.

[0056] To improve the viewing angle limitation caused by fixed channels in related methods, in some embodiments, the determined image data may include image data of multiple channels. In step S200, performing an image rendering operation using the determined image data may include: in response to a selection operation for one of the channels, calling the image data of the corresponding channel for image rendering.

[0057] In some embodiments, the data for each channel can correspond to a separate camera (e.g., channel 1 corresponds to the head camera, channel 2 corresponds to the left-hand camera). When storing data, it can be associated by channel number and timestamp. For example, a user can select a target channel (e.g., switching from "head camera" to "left-hand camera") using a drop-down menu or button in the visualization interface. Subsequently, based on the current playback frame (e.g., frame 50) and a channel selection (e.g., switching to "left-hand camera"), the image data for the corresponding channel can be retrieved from the storage system, etc., for image rendering. The rendered image results are then displayed in the first display area. For example, in a scenario where a robot is reviewing a "cup grabbing" action, the first display area can default to displaying the head camera image, showing the robot approaching the cup on the table. The user can simultaneously view the motion state rendering results (e.g., joint angle curves) displayed in the second display area to confirm that the robot's motion trajectory is normal. If the user finds that the grasping details are unclear from the head perspective, they select "left-hand camera" from the drop-down menu. The system automatically loads the left-hand camera image at the same moment (e.g., frame 50), showing the contact point between the gripper and the cup. Combined with motion state rendering results (such as joint angle curves), users can identify insufficient gripping force due to insufficient joint angle. The robot then corrects its motion, and the user switches back to the head camera to confirm the robot's overall posture.

[0058] It can be understood that since the determined image data includes image data of multiple channels, in the process of performing the image rendering operation, the image data of a corresponding channel can be called for image rendering in response to the selection operation for one of the channels. Therefore, in some embodiments, the present method can realize "multi-channel integration and dynamic switching of channels", which is a basic function for realizing efficient multimodal data analysis.

[0059] It can be seen that in some embodiments, it is possible to quickly switch perspectives without exiting the current visualization interface, thereby improving the problem of repeatedly switching software in related methods. Different camera perspectives can be quickly switched to achieve multi-perspective comparison, making it easier to discover data anomalies (such as local occlusion, sensor failure). When switching channels, other data (such as motion state rendering results) can still be updated based on the same current playback frame to ensure multimodal data consistency. In addition, the first display area can display the image rendering results obtained by the image data of the high-frequency channel (such as the head channel) by default, and the image data of other channels can be loaded only when the user selects through a drop-down box, etc., thereby significantly reducing the amount of image requests (such as from 90 requests / second to 30 requests / second), thereby improving the user experience.

[0060] Because related data visualization methods only display the joint angle values ​​for the current playback frame, and the data for each joint is scattered across different interfaces, it can be difficult to detect unusual patterns in robot joint motion. Even if these methods draw all joint angle curves at once, users still need to manually zoom / pan to locate the current frame and compare the timeline to guess the correspondence between curve points and frames, which is cumbersome and prone to misinterpretation.

[0061] In order to intuitively display the changing trend of the robot's joint angle over time and make up for the limitations of single frame data, in some embodiments, in step S300, the determined motion control data may include joint angle data from the first specified frame to the current playback frame, and the process of performing motion state rendering operations using the determined motion control data may include: generating line chart data based on the joint angle data from the first specified frame to the current playback frame, and drawing a line chart using the line chart data.

[0062] In some embodiments, based on a user-set first designated frame (e.g., frame 1) and a currently playing frame (e.g., frame 50), all joint angle data within this time interval can be extracted from the motion control data. For example, the angle values ​​per second of the robot's six joints (e.g., joints 1 to 6) from frame 1 to frame 50 can be extracted.

[0063] It is understandable that line graph data can be a structured time series data. Line graph data can describe the numerical sequence of the angles of each joint of the robot changing over time. Line graph data can provide the original data for chart rendering (such as drawing a line graph). Among them, the process of "generating line graph data" can be: classifying the processed data by joint, and generating an independent curve data set for each joint; using relevant technologies to automatically analyze the curve features of the curve data set, marking relevant key points (such as extreme points, mutation points and over-limit points) to generate line graph data. The specific generation process of line graph data will not be repeated here.

[0064] It is understood that the line graph derived from the line graph data can essentially be a chart that visualizes the line graph data. The horizontal axis of the line graph can be a time / frame index, and the vertical axis can be a joint angle value. The multiple lines in the line graph correspond to the angle change curves of different joints. The line graph can intuitively display movement trends and quickly identify whether the joint movement is smooth (for example, whether the curve shows a sudden change, indicating a possible mechanical failure). The line graph can also be used to overlay multiple joint curves to verify movement coordination. The line graph can also be used in conjunction with the playback controller described above. For example, clicking a point on the line graph can jump directly to the corresponding frame and view the image rendering results at that moment and the model rendering results below. The line graph can be drawn by setting the desired coordinate axes in the same chart area and plotting the angle changes of each joint using curves of different colors. The chart area can also include a current frame mark. The current frame mark can be used to highlight a vertical mark (such as the 50th frame position) on the horizontal axis to indicate the playback progress. The line graph can be drawn using relevant technologies, and the specific generation process will not be detailed here.

[0065] For example, in a robot assembly action review scenario, if the line graph shows that the angle of "Joint 3" does not change between frames 120 and 130, but other joints move normally, the user can infer that the possible cause is that the motor of Joint 3 is stuck / the control signal is lost. The user can also perform the following verification steps: click on the 125th frame of the line graph, and simultaneously view its image area (such as whether Joint 3 is blocked by foreign objects) and the model rendering results displayed in the third display area below (such as checking whether Joint 3 is in the extreme position). At the same time, when the user drags the progress bar of the playback controller to the Nth (N is a positive integer) frame, the line graph in the second display area can automatically highlight the corresponding data points of the Nth frame. The abnormal points on the line graph can be quickly located to the image rendering results displayed in the first display area and the model rendering results displayed in the third display area below, achieving triple verification of "curve, vision and space".

[0066] It can be seen that in some embodiments, line graph data can be dynamically generated, and only a subset of data (rather than all data) from the first specified frame (such as the 100th frame) to the current playback frame (such as the 125th frame) is maintained during playback, thereby reducing memory usage. Secondly, by converting discrete joint angle data into intuitive timing curves, the movement patterns are revealed. In addition, by automatically marking feature points, movement defects (such as jitter, over-limit) can be quickly located to improve the user's analysis efficiency. In addition, the above embodiments can also support the same-screen comparison of multiple joint curves to optimize motion planning.

[0067] Related data visualization methods usually rely only on two-dimensional data (such as camera images) and lack spatial mapping capabilities, making it difficult to determine whether the robotic arm has collided with the environment, whether the posture of the end effector is reasonable, etc.; single-view images may not reflect the real spatial relationship due to occlusion or lighting (such as the robotic arm being blocked by its own parts); in addition, if there is a suspected problem with the action, the physical robot needs to re-perform the action verification, which is time-consuming and may damage the equipment.

[0068] In order to improve the problem that the 2D view cannot present the spatial posture, in some embodiments, the visualization interface may further include a third display area. In step S300, the determined motion control data may include the joint angle data of the current playback frame. Figure 2 , Figure 2 FIG. 4 is a flow chart of another data visualization method provided by an embodiment of the present application. The method may include steps S100 to S400.

[0069] Step S400: Based on the joint angle data of the current playback frame, the robot model of the robot is driven to perform posture rendering, and the model rendering result is displayed in the third display area.

[0070] It is understandable that the order of step S200, step S300 and step S400 can be interchanged. Step S200, step S300 and step S400 can also be executed simultaneously.

[0071] In some embodiments, in step S400, the joint angle data for the current playback frame can be extracted from a storage system, etc. Subsequently, the robot's URDF (Unified Robot Description Format) model can be loaded. The URDF model can define the robot's three-dimensional structure, joint hierarchy, and motion constraints. For example, a URDF-loader (Unified Robot Description Format Loader) can be used to load and render the robot model. During playback, the joint angle data corresponding to the current playback frame can be used through the "Set Joint Values" interface provided by the URDF-loader to drive the robot model frame by frame for pose rendering. Other related methods can also be used for pose rendering, and the specific process of the robot model and pose rendering is not specifically limited here. The user can freely rotate, scale, and translate the rendered model and freely adjust the viewing angle, such as switching from a top-down perspective to a side view to check the clearance between the robot and the workbench. The third display area can have a cross-sectional viewing function. For example, when perspective mode is enabled, the internal transmission structure of the rendered model can be displayed to check the meshing state of the gears. The third display area can also have a pose comparison function. For example, the theoretical posture (such as a semi-transparent model) and the actual posture are superimposed and displayed to show the degree of difference between the theoretical posture (semi-transparent model) and the actual posture through different colors, such as blue for normal and red for deviation greater than 5 mm.

[0072] For example, in the welding robot motion verification scenario, when the current playback frame is the 10th frame, the robot is in a standby posture, and the 3D model in the third display area shows that all joints are straight and the welding gun is vertically upward; when the current playback frame is the 30th frame, the robot's joint 1 rotates to 60° and joint 4 bends to 90°. The above 3D model shows that the tip of the welding gun accurately reaches the starting point of the weld, and the system automatically detects that the distance from the tooling fixture is less than 2mm; when the current playback frame is the 45th frame (critical welding point), the robot's joint 2 has a high-frequency jitter of ±1°, and the 3D model in the third display area shows that the welding gun vibrates slightly. Combined with the line graph in the second display area, it is confirmed that the servo motor is overheating; when the current playback frame is the 60th frame (welding is completed), all joints of the robot return to a safe position, and the 3D model in the third display area shows that the end effector is fully retracted, and the trajectory playback verifies that there is no self-collision.

[0073] It is understandable that step S400 can convert abstract numerical data (such as the above-mentioned joint angle data) into intuitive spatial representation, allowing engineers to analyze robot movements like "operating a three-dimensional holographic projection", greatly reducing the threshold for motion planning debugging and fault diagnosis.

[0074] In some embodiments, the user can view the image rendering results of the first display area (such as the head camera image in a two-dimensional perspective), the motion state rendering results of the second display area (such as the above-mentioned line graph), and the model rendering results of the third display area (such as the above-mentioned 3D model in a freely rotating perspective) in the same visualization interface. The model rendering results of the third display area can be used to intuitively determine whether unnecessary detours and exceeding the workspace occur. The user can freely rotate the model rendering results (such as the 3D model) in the third display area and observe them from any perspective, such as top, side, etc.

[0075] In some embodiments, when the current playback frame changes, the model rendering result of the third display area (such as a 3D model with a freely rotating perspective) is automatically updated to the corresponding posture, forming a four-dimensional linkage of "timeline, image, curve and spatial model".

[0076] As can be seen, in some embodiments, the robot's three-dimensional configuration and motion trajectory can be intuitively displayed, with support for free perspective adjustment and zooming to view details, allowing for the detection of potential collisions or configuration anomalies. In other words, the upgrade from purely numerical analysis to spatially aware analysis overcomes the "blind men and elephant" dilemma inherent in two-dimensional audit tools. This represents a crucial leap forward in embodied intelligent data visualization, moving from "usable" to "efficient and reliable."

[0077] Because related data visualization methods generally require frame-by-frame requests, resulting in more than 30 HTTP requests (data packets used to request specific resources) per second, directly loading the original image data may cause the browser to crash, and data decoding may also block the main thread, resulting in a drop in frame rate.

[0078] In order to improve the problem of high-frequency image loading jamming in related data visualization methods, in some embodiments, the process of determining the image data used for visualization of the current playback frame may include: downloading an image data packet that has undergone multiple compression processes from an object storage system; decompressing the downloaded image data packet and storing it in a local database; and reading the image data corresponding to the current playback frame from the local database.

[0079] It's understandable that object storage systems can be considered long-term storage and distribution centers. They provide highly reliable, low-cost storage for massive amounts of data. They can store multiply compressed image data packets in time-based chunks. Downloading multiply compressed image data packets from object storage systems can alleviate network bottlenecks, for example, reducing 30 HTTP requests per second to just one, thereby mitigating network storms caused by frame-by-frame loading. Furthermore, they can save bandwidth, for example, by reducing the amount of data transmitted by over 80% through multiple compression processes.

[0080] In some embodiments, pre-processed compressed data packets, i.e., the aforementioned multi-compressed image data packets, can be downloaded from the object storage system based on the time period of the currently played frame (e.g., the 50th frame belongs to the data packet at the 2nd second) or a time index. The image data packets can be files formed by multi-compressing image frames within each specified time period (e.g., 1 second). It is understood that the multi-compressed image data packets are highly compressed (e.g., their size can be only 10%-20% of the original uncompressed data).

[0081] It is understandable that the local database is used for browser-side caching. The local database can achieve millisecond-level data reading, reducing repeated network requests. The local database can also support offline review (such as accessing cached data when there is no network connection). The local database can store decompressed single-frame data. It can be available offline, such as after a single download, subsequent playback does not require repeated requests. In addition, it can also achieve accelerated reading, for example, the local database access speed can be 10 to 100 times faster than the network request.

[0082] In some embodiments, the download task can be handed over to background threads such as Web Worker (Web Worker API, browser multi-threading) through the main thread to reduce the occurrence of blocking the main thread. The downloaded image data packet that has been compressed multiple times can be directly decompressed using relevant algorithms. The decompressed image data packet can be named and stored according to the frame index. The local database can use the browser's built-in IndexedDB (Indexed Database API, index database) or other database. The local database can support structured storage. The local database can automatically clean up older data by establishing an LRU (Least Recently Used) cache mechanism. For example, the decompressed 30 frames of image are stored in the local database, and the subsequent playback of the 2nd second content can be directly read locally without the need to re-download.

[0083] It is understandable that reading the image data corresponding to the currently playing frame from the local database can significantly reduce the latency of local reading, thereby ensuring smooth playback. In addition, the target frame can be displayed immediately when the user drags the progress bar, that is, accurate frame skipping is supported.

[0084] In some embodiments, the image data corresponding to the current playback frame can be calculated based on the current playback frame (such as the 83rd frame is the 23rd frame at the 3rd second). For example, the 83rd frame or the 23rd frame at the 3rd second can be read from the local database. The user can also quickly jump to the 5th minute and 30th second (the 9900th frame) through the corresponding operation, and the system can locate the data packet at the 330th second (9900÷30). The 0th frame at the 330th second is directly read from the local database (9900%30=0). Afterwards, the corresponding rendering preparation is performed.

[0085] It can be seen that in some embodiments, the performance bottleneck of real-time analysis of large-scale robot vision data can be improved through multiple compressions and a local cache architecture.

[0086] In some embodiments, the image data packets may be organized by time period. The image data packet compression process may include: performing file-level merging on multiple image frames of the robot within the corresponding time period to achieve a primary compression; and performing a secondary compression on the files obtained by the primary compression to obtain the image data packet.

[0087] For example, multiple image frames of a robot within a corresponding time period (e.g., 1 second) can be merged into a single file to achieve primary compression. Those skilled in the art can dynamically adjust the length of the corresponding time period based on the actual scenario. The primary compression can utilize a lossless compression algorithm. Subsequently, the file obtained by the primary compression can be subjected to secondary compression. The secondary compression can be lossy compression. Alternatively, the primary compression can utilize a lossy compression algorithm, and the secondary compression can utilize a lossless compression algorithm. Alternatively, both the primary and secondary compression can utilize lossless compression algorithms, or both the primary and secondary compression can utilize lossy compression algorithms. For example, the secondary compression can utilize the Gzip algorithm (a lossless data compression algorithm) or the Brotli algorithm (an open-source, lossless compression algorithm). The compression methods used for the primary and secondary compression are not limited herein. Those skilled in the art can dynamically adjust the compression ratio of the secondary compression based on the actual scenario. For example, the compression ratio can be 60% for complex scenarios and 80% for simple scenarios. The image data packets obtained by the secondary compression can be stored in an object storage system.

[0088] It can be seen that in some embodiments, multiple image frames of the robot within the corresponding time period can be merged into a single file through the first level of compression, thereby significantly reducing the number of network requests. The merged file can be further compressed through the second level of compression. The object storage system can be divided into blocks by time (such as one image data packet every 10 seconds), and can thus support corresponding range requests. When needed, the image data packet that has undergone multiple compression processes is downloaded to the local database, and the image data for visualization of the current playback frame is read on demand. In addition, the image data packets required subsequently are downloaded from the object storage system, thereby reducing the blocking of the main thread and reducing the load on the main thread. The local database stores the decompressed data, and subsequent playback directly reads the local database, which can improve the smoothness of playback.

[0089] In related data visualization methods, labeling usually requires finding the target frame in a visualization area (such as a video playback window), manually recording the frame number, and then switching to a form area to enter the frame number and fill in a description. Users need to repeatedly switch between the two independent processes of "visual positioning" and "form input", resulting in an excessively long operation path. In addition, when relying solely on the form to enter the frame number, it is easy for the labeling to shift due to input errors or visual misjudgments (such as mistakenly labeling the 120th frame as the 125th frame).

[0090] In order to improve the inefficient interaction problem during the robot data annotation process, in some embodiments, the visual interface may further include an annotation alignment area. The method may further include step S500.

[0091] Step S500: Displaying at least one first marker in the annotation alignment area, the at least one first marker including an action step marker and / or a keyframe marker; in response to a selection operation on the first marker, performing an operation of aligning the first marker to the currently playing frame, and associating the first marker with the frame index information of the currently playing frame and saving it. The first marker may be a graphic mark or a text label, and the selection operation may include a drag operation and / or a double-click operation.

[0092] In some embodiments, the visualization interface further includes an integrated annotation alignment area. At least one first identifier is displayed in the annotation alignment area. The at least one first identifier includes an action step identifier and / or a key frame identifier. The action step identifier can be used to display the step name. Figure 1bThe action step identifier 105 can be in the form of a graphical mark. For example, a colored triangle can be used to represent the grasping step. The action step identifier 105 can also be represented in the form of a text label. For example, the step name can be displayed in a floating manner (such as "Step 2: Parts Assembly"). It is understood that an action step can correspond to a continuous time period, and the action step identifier 105 can be used to indicate the starting time, ending time, and intermediate time of the action step. In this case, the action step identifier 105 can be a triangle or rectangle mark. Alternatively, the action step identifier 105 can be used to identify the time period corresponding to the action step. In this case, the action step identifier 105 can be a bar mark or a floating mark. Aligning the action step identifier 105 with the current playback frame can be regarded as aligning the current playback frame to the middle time of the corresponding time period of the action step. The user can add corresponding notes to the action step corresponding to the action step identifier 105 through the form annotation sub-area 107 within the visual interface 100. The key frame identifier 106 can be used to display key parameters. The key frame identifier 106 can be in the form of a graphical mark, for example, a circular icon can be used to represent the maximum pressure point. The key frame identifier 106 can also be represented in the form of a text label, for example, a floating display of key parameters (such as "torque peak: 5.2N·m"). The user can enter the relevant text information of the action step identifier 105 and the key frame identifier 106 through the form annotation sub-area 107 of the visual interface 100. It can be understood that the key frame here can be preset or manually judged and newly created by the user (such as annotator). Key frames are generally used to describe the "specific moment" and "important event points" in the specified task. The key frames in each specified task may be different. For example, the above-mentioned action steps can be action stages such as "grabbing the water bottle", "moving", and "putting down", and the "key frame" is the frame of data that the user temporarily decides to record when observing the video, such as the data frame corresponding to the moment the robot lets go of its hand and puts down the water bottle. See Figure 1bAfter the user determines the "key frame" to be recorded, the user can click the "Pause" button on the playback controller 101, and the corresponding image will stop at the "key frame" (such as frame 1983). Afterwards, the user can click the key frame identifier 106, and the corresponding area will pop up "New Key Frame". The user can enter "Let go and put down the water bottle" through the form annotation sub-area 107 in the visual interface 100. After that, the system automatically adds a new key frame identifier 106 to the timeline on the visual interface 100. The user drags the key frame identifier 106 until it aligns with the position of the above-mentioned frame 1983. Finally, the system automatically corresponds the key frame "Let go and put down the water bottle" to the 1983rd frame. In addition, the user can also add corresponding remarks to the key frame "Let go and put down the water bottle" corresponding to the key frame identifier 106 through the form annotation sub-area 107 of the visual interface 100, such as "The water bottle is about to be released, the end position is (x:0.6, y:0.2, z:0.8), and the joint angle J5 reaches the maximum value."

[0093] Those skilled in the art can determine the number and category of the first identifiers in the annotation alignment area according to actual needs. After the user pauses playback, he can directly drag the first identifier in the annotation alignment area to the current frame mark (or double-click to automatically align), and the system can automatically associate and save the first identifier with the frame index information of the currently playing frame. Among them, the frame index information may refer to the integer value corresponding to the currently playing frame in the playback controller (such as the 120th frame). The purpose of "associating and saving the first identifier with the frame index information of the currently playing frame" is to bind and store the user's annotated business intention (such as "this is the midpoint of the grabbing action") with a specific data location (such as the 120th frame).

[0094] For example, in the scenario of robot welding task review, the annotation alignment area can display three action step identifiers (for example, the yellow triangle symbol represents the positioning weld point, the red triangle symbol represents the arc welding, and the green triangle symbol represents the arc end detection) and two key frame identifiers (for example, the first red dot represents the point of maximum welding current, and the second red dot represents the weld completion point). For another example, in the scenario of robot assembly task annotation, when the data is played to the 200th frame, the user finds the moment when the screwdriver touches the screw head, and drags the "screw alignment" action step identifier to the current playback frame. The system automatically saves the association, such as: "The label is screw alignment, and the frame index is frame 200". When you click the "screw alignment" action step identifier again, you can jump directly to the 200th frame to check whether the screwdriver posture is compliant. For another example, in the scenario of surgical robot training data annotation, the doctor can directly drag the "incision start" action step marker to the precise frame where the knife tip contacts the tissue in the endoscope image. When double-clicking to add the "bleeding point" keyframe marker, the system automatically associates the instrument pressure data, 3D instrument posture and patient vital signs at that time to build a complete surgical action knowledge graph.

[0095] It is understood that the purpose of step S500 is to transform abstract action steps and key events into intuitive graphical elements. The layout of the icons also allows for a quick understanding of the task phases (e.g., preparation period, execution period, and completion period). Furthermore, a visual operating handle can be provided for subsequent annotation and review operations.

[0096] It can be seen that in some embodiments, by forcibly binding the first identifier and the frame index, an accurate mapping between user operations and data locations can be established, thereby achieving accurate backtracking. For example, during subsequent review, you can click on the action step identifier indicating "start crawling", and the system will automatically jump to the 120th frame to restore the complete scene at the time of annotation. The annotated data can be directly converted into the structured format required for model training. The annotations of different users are based on the same frame index system, which can reduce confusion caused by differences in descriptions. In addition, it can support jumping frames from the first identifier, or retrieving the first identifier from the frame, thereby accelerating problem location and data auditing. In other words, the above embodiments can form a closed loop of "visualization, interaction, and association", improve annotation efficiency, improve annotation accuracy, and improve the quality of annotation data.

[0097] Related data visualization methods often use sliders to control playback progress. However, due to the high frame rate of data, the minimum slider step size corresponds to multiple frames, making it difficult to accurately locate a single frame. This further forces users to repeatedly fine-tune the slider or use frame-by-frame buttons, which is time-consuming and can easily lead to missing the target frame. Furthermore, the progress bar usually does not mark the position of the current playback frame, making it difficult for users to quickly determine the offset between the slider and the target frame.

[0098] In order to achieve frame-level precise positioning and improve the problem of coarse-grained control of the progress bar in the related methods, in some embodiments, the method may further include step S600.

[0099] Step S600: Displaying a second marker in the playback controller, the second marker is used to indicate the position of the current playback frame; in response to a drag operation on the second marker, updating the current playback frame to the playback frame corresponding to the dragged position of the second marker, wherein the second marker is a line or a graphic mark.

[0100] In some embodiments, a second marker can be displayed in the playback controller. The user can directly drag the second marker to the target position. The second marker can be a marking line, such as a red vertical line. The second marker can also be a graphic mark, such as a triangle. The precise frame number can also be displayed on the second marker. It is understandable that the user can quickly jump to any desired position by dragging the second marker. For example, when the user wants to see the key step of the robot adding salt, click the second marker (such as a red marking line) with the mouse and drag it to the right to the middle of the video. After the user releases the mouse, the video immediately jumps to the middle time point. At this time, the first display area can display a real-time picture of the robot picking up the salt tank, the 3D model on the third display area can synchronously display the robot's bending posture, and the second display area can display the corresponding joint angle curve graph and corresponding data. In response to the drag operation on the second marker, the current playback frame can be updated to the corresponding playback frame of the dragged position of the second marker. In addition, the system can update the current playback frame at frame granularity.

[0101] It is understood that by displaying the second marker in the playback controller, the draggable second marker (such as a red mark) can be displayed in real time on the playback progress bar. The above embodiment can also establish a visual association with multiple views (such as related views in the first, second, and third display areas), such as when the second marker is moved, all views are synchronously highlighted.

[0102] It is understandable that in response to a drag operation on the second marker, a discrete frame jump (e.g., dragging from the 100th frame to the 125th frame) can be automatically triggered, thereby reducing the continuous value error caused by the use of a slider in the relevant visualization method. For example, when analyzing a scene of a robot assembly action, the total length of the progress bar represents a 2-minute task duration (e.g., a total of 3600 frames), and the second marker (e.g., a red line) can indicate in real time that the current playback is at the 1250th frame. The right side of the second marker (e.g., a red line) can display the available status icons of the three camera channels of the frame.

[0103] In some embodiments, in response to the drag operation on the second identifier, the aforementioned Web Worker (Web Worker API, browser multi-threading) can preload adjacent frame data to improve the problem of lag after jumping.

[0104] As can be seen, in some embodiments, in response to a drag operation directed at the second marker, a discrete frame jump (e.g., dragging from frame 100 to frame 105) can be automatically triggered, thereby reducing the continuous value error caused by the use of a slider in related visualization methods. In other words, the multiple fine-tuning required in related visualization methods (e.g., 3 to 5 operations) is reduced to a single drag operation, significantly shortening the target frame positioning time. In addition, the above embodiments can reduce jump delays, thereby meeting real-time interaction requirements as much as possible.

[0105] In relevant data visualization methods, action steps are usually displayed separately in text form, such as "Step 1: Grab the object, the estimated time is 2 to 5 seconds". Users need to construct the timeline mapping relationship in their minds and repeatedly compare the step description with the playback progress, which can easily lead to misunderstandings. In addition, the expected step time range set when the task is created (such as "Step 1 should be completed in 2 to 5 seconds") is not intuitively reflected in the interface, making it difficult to discover the deviation between actual data and expectations.

[0106] In order to intuitively display the time range of the action steps and improve the tedious problem of manually calculating the time period, in some embodiments, the method may further include step S700.

[0107] Step S700: Based on the set time period corresponding to the action step, a marking operation is performed on the time axis of the playback controller using a third identifier.

[0108] In some embodiments, a third identifier can be added below the timeline of the playback controller. The set time period can be determined based on the preset task action information. The third identifier can be a line segment or bar mark located below the timeline, or a line or a block graphic mark in the form of a floating layer covering the timeline for indicating the start and end time range. The third identifier can also be in other forms, which are not specifically limited here. The length and position of the third identifier can correspond to the start and end time of the preset action step. For example, if the "grab object" step is preset to the 2nd to 5th seconds, a green line segment indicating the 2nd to 5th seconds will be displayed.

[0109] In some embodiments, the user can click the third marker to quickly jump to the step start frame, which can help quickly locate the problem interval.

[0110] It is understandable that the purpose of step S700 is to convert the abstract text description into a concrete time range marker (such as the third marker) to reduce the user's cognitive load. In addition, if the actual action exceeds the range indicated by the third marker (such as the step not being completed before the 5th second), the third marker can indicate an abnormality.

[0111] As can be seen, in some embodiments, the step alignment check time can be significantly shortened, improving the efficiency of review or annotation. In addition, it is also more conducive to the quality monitoring of the review or annotation process. For example, the detection rate of deviations between the preset time and the actual time can be increased to more than 95%.

[0112] The above is an explanation of the data visualization method. This embodiment of the application also provides a data visualization system. Figure 3 , Figure 3Schematic diagram of a data visualization system provided by an embodiment of the present application. The system may include: a visualization interface 100, a data determination module 200, an image processing module 300 and a state processing module 400. Figure 4 , Figure 4 is a schematic diagram of another visualization interface provided by an embodiment of the present application. The visualization interface 100 may include a playback controller 101, a first display area 102, and a second display area 103. The data determination module 200 may determine, based on the current playback frame in the playback controller 101, image data corresponding to the robot for visualization of the current playback frame and motion control data related to the robot state. The image processing module 300 may be used to perform an image rendering operation using the determined image data and display the image rendering result in the first display area 102. The state processing module 400 may be used to perform a motion state rendering operation using the determined motion control data and display the motion state rendering result in the second display area 103.

[0113] It should be noted that the relevant content of each module of the data visualization system here is basically the same as the relevant content of the data visualization method mentioned above. The technical effects achieved by this data visualization system are also basically the same as those of the data visualization method mentioned above. Therefore, it will not be repeated here.

[0114] In some embodiments, the determined motion control data may include joint angle data of the current playback frame. Figure 5 , Figure 5 This is a module diagram of another data visualization system provided by an embodiment of the present application. The system may also include a model processing module 500. The model processing module 500 can be used to drive the robot model of the robot to perform posture rendering based on the joint angle data of the current playback frame, and display the model rendering result in the third display area. Figure 6 , Figure 6 100 is a schematic diagram of another visualization interface provided by an embodiment of the present application. The visualization interface 100 may include a playback controller 101 , a first display area 102 , a second display area 103 and a third display area 104 .

[0115] It should be noted that the relevant content of each module of the data visualization system here is basically the same as the relevant content of the data visualization method mentioned above. The technical effects achieved by this data visualization system are also basically the same as those of the data visualization method mentioned above. Therefore, it will not be repeated here.

[0116] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the data visualization methods in the above embodiments is implemented.

[0117] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements any one of the data visualization methods in the above embodiments.

[0118] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the embodiments of the present application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0119] See also Figure 7 , Figure 7 This is a structural block diagram of a computer device provided in an embodiment of the present application.

[0120] An embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above-mentioned data visualization methods when executing the computer program.

[0121] The embodiments of the present application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0122] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected via an internal connection path.

[0123] The memory 110 is used to store computer programs. In some implementations, the computer programs may include codes for implementing the methods of the embodiments of the present application.

[0124] The processor 120 is configured to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output data such as operation results. In some implementations, when the solutions of the embodiments of the present application are implemented through software or firmware, the computer program for implementing the solutions of the embodiments of the present application may be stored in the processor 120 and executed by the processor 120.

[0125] The memory 110 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes a random access memory (RAM), a cache memory and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120 so that the processor 120 implements the steps of any of the above methods.

[0126] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or the processor 120 may be any conventional processor.

[0127] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 120 or by instructions in the form of software. The method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor 120. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0128] In some implementations, in addition to the hardware units described above, the computer device may also include software modules, where the software modules may be, for example, an operating system, a basic input and output system (BIOS), application software, etc.

[0129] An operating system manages the hardware and / or software resources of a computer device and is the core and cornerstone of the computer. It handles basic tasks such as managing and allocating memory, prioritizing the supply and demand of system resources, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide an interface for users to interact with the system.

[0130] The BIOS is used to run hardware initialization during the power-on boot phase and provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display the processor temperature and execute functions such as adjusting temperature protection strategies.

[0131] Application software, also known as an application program, is software written for a specific user purpose. It is a major category of computer software. For example, application software might be a program used for power control, temperature management, and other purposes.

[0132] It should be understood that the specific examples in this application are only intended to help those skilled in the art better understand the implementation methods of this application, rather than to limit the scope of protection of this application.

[0133] It can be understood that in various implementations of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the present application.

[0134] It can be understood that the various embodiments described in this application can be implemented individually or in combination, and this application is not limited to this.

[0135] Unless otherwise indicated, all technical and scientific terms used in this application have the same meaning as those commonly understood by those skilled in the art in the art of this application. The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit the scope of this application. The singular forms "a," "above," and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0137] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described embodiments may refer to the corresponding processes in other embodiments and will not be repeated here.

[0138] In the embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the objectives of the technical solutions of this application.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0141] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0142] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data visualization method, characterized in that: Applied to a data visualization system, wherein a visualization interface in the system includes a playback controller, a first display area, and a second display area, the method includes: Determining, based on the current playing frame in the playing controller, image data corresponding to the robot for visualization of the current playing frame and motion control data related to the robot state; performing an image rendering operation using the determined image data, and displaying the image rendering result in the first display area; A motion state rendering operation is performed using the determined motion control data, and a motion state rendering result is displayed in the second display area.

2. The data visualization method according to claim 1, characterized in that The determined image data includes image data of multiple channels, and the process of performing an image rendering operation using the determined image data includes: In response to a selection operation on one of the channels, image data of the corresponding channel is called for image rendering.

3. The data visualization method according to claim 1, characterized in that The determined motion control data includes joint angle data from the first designated frame to the current playback frame, and the process of performing the motion state rendering operation using the determined motion control data includes: Line graph data is generated based on the joint angle data from the first designated frame to the current playing frame, and a line graph is drawn using the line graph data.

4. The data visualization method according to claim 1, characterized in that The determined motion control data includes joint angle data of the current playback frame, the visualization interface further includes a third display area, and the method further includes: Based on the joint angle data of the current playing frame, the robot model of the robot is driven to perform posture rendering, and the model rendering result is displayed in the third display area.

5. The data visualization method according to claim 1, characterized in that: The process of determining the image data used for visualization of the current playback frame includes: Download the image data package that has been compressed multiple times from the object storage system; Decompress the downloaded image data packet and store it in a local database; The image data corresponding to the current playing frame is read from the local database.

6. The data visualization method according to claim 5, characterized in that: The image data packets are organized by time period, and the compression process of the image data packets includes: Merging multiple image frames of the robot within a corresponding time period at a file level to achieve one-time compression; The file obtained by the primary compression is compressed twice to obtain the image data packet.

7. The data visualization method according to claim 1, characterized in that: The visual interface further includes a marking alignment area, and the method further includes: Displaying at least one first identifier in the annotation alignment area, wherein the at least one first identifier includes an action step identifier and / or a key frame identifier; In response to a selection operation on the first identifier, performing an operation of aligning the first identifier to a current playback frame, and associating and storing the first identifier with frame index information of the current playback frame; The first identifier is a graphic mark or a text label, and the selection operation includes a drag operation and / or a double-click operation.

8. The data visualization method according to claim 1, characterized in that: The method further comprises: Displaying a second identifier in the playback controller, where the second identifier is used to indicate the position of the current playback frame; In response to a drag operation on the second marker, updating the current playback frame to a playback frame corresponding to the dragged position of the second marker; The second mark is a marking line or a graphic mark.

9. The data visualization method according to claim 1, characterized in that: The method further comprises: Based on a set time period corresponding to an action step, using a third identifier to perform a marking operation on the time axis of the playback controller; the set time period is determined according to preset task action information; The third mark is a line segment or a bar mark below the time axis, or a marking line or a floating block graphic mark covering the time axis for marking the start and end time range.

10. A data visualization system, characterized in that: The system comprises: A visual interface, comprising a playback controller, a first display area, and a second display area; a data determination module, which determines, based on the current playback frame in the playback controller, image data corresponding to the robot for visualization of the current playback frame and motion control data related to the robot state; an image processing module, configured to perform an image rendering operation using the determined image data, and display the image rendering result in the first display area; The state processing module is configured to perform a motion state rendering operation using the determined motion control data, and display a motion state rendering result in the second display area.

11. The data visualization system according to claim 10, characterized in that The determined motion control data includes joint angle data of the current playback frame, the visualization interface further includes a third display area, and the system further includes: The model processing module is used to drive the robot model of the robot to perform posture rendering based on the joint angle data of the current playback frame, and display the model rendering result in the third display area.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images