Data processing method, system and apparatus

Abstract

Embodiments of the present specification provide a data processing method, system and device, wherein the data processing method comprises: loading a virtual model, determining static scene data carrying a scene identifier based on the virtual model, and determining dynamic data associated with the scene identifier; calling an image generation module, generating initial image data containing at least two types of data based on an image generation unit set, the static scene data and the dynamic data; and constructing target image data based on the image generation unit set, the dynamic data, the scene identifier and the initial image data.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of simulation image technology, and in particular to data processing methods, systems and devices. Background Technology

[0002] In the field of simulation or data processing for virtual scene image generation, it is typically necessary to generate corresponding image data based on the interaction state between the static environment and dynamically moving objects in the scene, for use by downstream perception, control, or analysis modules. In existing technologies, the image generation process involves repeated loading of static scenes. When multiple image output terminals independently load scene description files, on the one hand, the repeated transmission of static scene data wastes bandwidth; on the other hand, inconsistencies in the model versions loaded by different output terminals can lead to deviations in the scene state corresponding to the image data generated at the same time, affecting the accurate judgment of the scene state by downstream modules. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising: Load a virtual model, determine static scene data carrying scene identifiers based on the virtual model, and determine the dynamic data associated with the scene identifiers; The image generation module is invoked to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data. Target image data is constructed based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data.

[0005] Optionally, determining the static scene data carrying scene identifiers based on the virtual model includes: Determine the robot model and environment model included in the virtual model; Extract robot data from the robot model description file of the robot model, and extract scene data from the environment model description file of the environment model; The static scene data is constructed based on the robot data, the scene data, the scene identifier corresponding to the virtual model, and the camera parameters.

[0006] Optionally, determining the dynamic data associated with the scene identifier includes: Determine the dynamic pose frame and state data corresponding to the virtual model; The dynamic data associated with the scene identifier is generated based on the dynamic pose frame and the state data.

[0007] Optionally, the construction of the image generation unit set includes: Candidate image generation units are determined based on the robot state table and / or camera enable configuration table; At least one image generation unit is determined from the candidate image generation units based on robot state information and / or camera state information, and the image generation unit set is constructed based on the at least one image generation unit.

[0008] Optionally, the invocation of the image generation module, based on the image generation unit set, generates initial image data containing at least two types of data from the static scene data and the dynamic data, including: Based on the dynamic pose frames in the dynamic data, the image generation units in the image generation unit set are combined to obtain the image rendering task. The image generation module is invoked to perform the image rendering task, and the color dataset and depth dataset are obtained. The initial image data is obtained by storing the color data from the color dataset into the color data region of the combined region, and storing the depth data from the depth dataset into the depth data region of the combined region.

[0009] Optionally, constructing the target image data based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data includes: The color data and the depth data are determined from the initial image data; The color data is converted into image color data, and the depth data is converted into a depth array; Image generation unit data is determined based on the image generation unit set, and simulation data is determined based on the dynamic data. The target image data is generated based on the image generation unit data, the simulation data, the image color data, and the depth array.

[0010] Optionally, before invoking the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the method further includes: Determine the static scene version information corresponding to the static scene data, and the dynamic version information corresponding to the dynamic data; If the static scene version information is consistent with the dynamic version information, the step of calling the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data and the dynamic data is executed. If the static scene version information is inconsistent with the dynamic version information, the target static scene data is obtained.

[0011] Optionally, after determining the dynamic data associated with the scene identifier, the method further includes: Determine the target frame rate and extract time information from the dynamic data; If, based on the target frame rate and the time information, it is determined that detection is generated through image data, the step of calling the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data is executed.

[0012] Optionally, the data processing method further includes: The data processing module is invoked to provide a data publishing entry to the image generation module, which is used to publish the target image data, and a data subscription entry is provided to the container module, which is used to subscribe to the target image data; The data processing module is used to send image data to the algorithm module in the container module according to the data sending rules.

[0013] Optionally, the data processing method further includes: Determine the data processing information and anomaly information associated with the target image data; The data processing information is stored, and an exception handling task is generated based on the exception information. The exception handling task is executed to complete the exception recovery.

[0014] According to a second aspect of the embodiments of this specification, a data processing system is provided, including a physical terminal, an image generation module, a data processing module, and an algorithm module, comprising: The physical terminal is used to load a virtual model, determine static scene data carrying scene identifiers based on the virtual model, and determine dynamic data associated with the scene identifiers; and send the static scene data and the dynamic data to the image generation module. The image generation module is used to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data; construct target image data based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data; and send the target image data to the data processing module through the data publishing entry. The data processing module is used to receive subscription information for the target image data from the algorithm module and send the target image data to the algorithm module.

[0015] According to a third aspect of the embodiments of this specification, a data processing apparatus is provided, comprising: The loading unit is configured to load a virtual model, determine static scene data carrying scene identifiers based on the virtual model, and determine dynamic data associated with the scene identifiers. The generation unit is configured to call the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data. The construction unit is configured to construct target image data based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data.

[0016] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0017] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0018] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0019] This specification provides a data processing method in one embodiment, which loads a virtual model, determines static scene data carrying scene identifiers based on the virtual model, and determines dynamic data associated with the scene identifiers. An image generation module is invoked to generate initial image data containing at least two types of data based on an image generation unit set, static scene data, and dynamic data. Target image data is constructed based on the image generation unit set, dynamic data, scene identifiers, and initial image data to ensure consistency in scene version among the initial image data output by each image generation unit, avoiding image anomalies caused by version mismatch between static scene and dynamic data, and reducing the burden of repeated transmission or loading of static scene data. Static scene data does not need to be repeatedly transmitted, reducing data retransmission overhead. Attached Figure Description

[0020] Figure 1This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 2 This is a schematic diagram illustrating data generation of a data processing method provided in one embodiment of this specification; Figure 3 This is a schematic diagram of data transmission for a data processing method provided in one embodiment of this specification; Figure 4 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification. Figure 5 This is a data processing flowchart of a data processing method provided in one embodiment of this specification; Figure 6 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification; Figure 7 This is a topology diagram of a data processing system provided in one embodiment of this specification; Figure 8 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification; Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0021] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0022] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0024] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0025] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0026] RGB images are a raster image format where each pixel is composed of values ​​from three color channels: red, green, and blue. They represent most colors visible in nature by mixing these three primary colors at different intensities. In robot simulation systems, RGB images simulate color visual information captured by a real camera, supporting downstream perception tasks such as target recognition and scene understanding.

[0027] A GPU graphics environment refers to the complete software stack that provides runtime support for a graphics processor, including graphics drivers, low-level graphics application programming interfaces (APIs), and related runtime library files. In containerized deployment scenarios, configuring a GPU graphics environment typically involves mapping the host machine's graphics card device file into the container and installing drivers and API runtime libraries that match the host machine's kernel version.

[0028] Graphics rendering API: refers to the standard programming interface between the application and the GPU hardware, used to convert 3D scene descriptions (including mesh geometry data, material parameters, texture maps and lighting information) into 2D screen pixels or off-screen buffer image data.

[0029] Physical WebSocket: A persistent, bidirectional communication channel based on the WebSocket protocol, specifically designed for transmitting physical state data in simulation systems. Unlike the traditional HTTP request-response model, WebSocket maintains a long-lived connection after it is established, allowing the server to continuously push dynamic pose data, simulation timestamps, and physical control commands to the client. This results in lower communication latency and less header overhead. In distributed simulation architectures, physical WebSocket connections are commonly used for real-time data exchange between the physical endpoint and the rendering or control endpoint.

[0030] An RGB-D image frame is a data unit containing both color and depth image data, acquired or generated at the same timestamp. The color image provides texture information about objects in the scene, while each pixel of the depth image stores the distance (usually in millimeters) from its corresponding spatial point to the image acquisition device. Both images maintain consistent spatial resolution, with a one-to-one correspondence between pixel positions, enabling downstream algorithms to map two-dimensional pixel coordinates to three-dimensional spatial points. RGB-D image frames typically also include metadata such as frame sequence numbers, simulation timestamps, and scene version identifiers to ensure time synchronization and scene consistency between multi-view data.

[0031] `host.docker.internal` is a special DNS domain name provided by the Docker container runtime for internal container processes to resolve and access the host machine's IP address. In Docker's default network mode, containers have their own independent network namespace and cannot directly access services running on the host machine via their local loopback address. The introduction of this domain name simplifies the communication configuration between the container and the host machine; developers can establish connections within the container using this domain name without explicitly obtaining the host machine's real IP address.

[0032] This specification provides a data processing method, and also relates to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0033] See Figure 1 , Figure 1 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0034] Step 102: Load the virtual model, determine the static scene data carrying the scene identifier based on the virtual model, and determine the dynamic data associated with the scene identifier.

[0035] Specifically, a virtual model refers to the model in the robot simulation image generation scene. Virtual modules include robot models and scene models. Scene identifiers are the scene version identifiers of the virtual scene, such as scene ID and scene name. Static scene data refers to static scene snapshots of the virtual model. Dynamic data refers to dynamic data published by the physical terminal, which will continuously publish dynamic data.

[0036] Based on this, in the robot simulation image generation scenario, when generating images, a virtual model is first loaded. Static scene data carrying scene identifiers is determined based on the virtual model, and a static scene snapshot of the virtual model is generated. This static scene snapshot carries a scene version identifier. Dynamic data carrying scene identifiers is continuously published from the physical terminal.

[0037] In practical applications, embodied robot software typically needs to receive RGB images, depth images, and camera calibration parameters consistent with the interface of a physical camera. Therefore, robot simulation systems need to generate images from multiple robot head or body cameras based on the physical scene. This embodiment provides an image generation and distribution scheme that does not require the container to run a GPU graphics environment, does not rely on a browser page, maintains synchronization with authoritative physical states, and can limit end-to-end camera latency.

[0038] Furthermore, considering that the virtual model includes both a robot model and an environment model, when constructing static scene data, data extraction can be performed separately for the robot model and the environment model. The specific implementation is as follows: The virtual model includes a robot model and an environment model; robot data is extracted from the robot model description file of the robot model, and scene data is extracted from the environment model description file of the environment model; static scene data is constructed based on the robot data, the scene data, the scene identifier corresponding to the virtual model, and the camera parameters.

[0039] Specifically, the robot model refers to the virtual robot in the robot simulation scene, and the environment model refers to the virtual scene in the robot simulation scene, i.e., the environment model. The environment model includes stationary objects such as the ground, walls, trees, buildings, and furniture. The robot model contains the robot's own geometric and physical description. The robot model description file is a file describing the geometric and physical properties of the robot itself within the robot model. The environment model description file is a file describing the properties of static objects within the environment model. Scene data includes, but is not limited to, the environment model's mesh, materials, textures, and lighting data. Robot data includes, but is not limited to, the robot's own mesh, materials, textures, geometric properties, and rigid body information. Camera parameters are independently configured and come from camera configuration files or sensor definition files in the simulation scene. Camera parameters include, but are not limited to, camera identifier, camera name, image width and height, field of view, near and far clipping planes, and data that can be used to construct intrinsic matrix, projection matrix, and distortion parameters. Static scene data generated from the virtual model does not depend on a specific graphics engine.

[0040] Based on this, the robot model and environment model included in the virtual model are determined. Robot data, such as mesh, material, texture, geometric properties, and rigid body information, is extracted from the robot model's robot model description file. Scene data, such as mesh, material, texture, and lighting, is extracted from the environment model's environment model description file. Camera identifiers, camera names, image width and height, field of view, near and far clipping planes, and camera parameters that can be used to construct intrinsic matrix, projection matrix, and distortion parameters are determined. Based on the robot data, scene data, scene identifiers corresponding to the virtual model, and camera parameters, static scene data independent of a specific graphics engine is constructed.

[0041] For example, in a robot simulation image generation scenario, the virtual model includes a robot model and an environment model. After loading the robot and environment models on the physical side, robot data such as mesh, material, texture, geometric properties, and rigid body information can be extracted. Scene data such as mesh, material, texture, and lighting can be extracted from the environment model. Camera parameters such as camera identifier, camera name, image width and height, field of view, near and far clipping planes, and parameters that can be used to construct intrinsic matrix, projection matrix, and distortion parameters can also be extracted. The physical side extracts the pure data content from the original model file, removes formatting information related to specific graphics rendering APIs, and forms a data list that can be recognized by any rendering engine, that is, static scene data that does not depend on a specific graphics engine.

[0042] In summary, static scene data is constructed based on robot data, scene data, scene identifiers corresponding to the virtual model, and camera parameters, without relying on a specific graphics engine. During physical runtime, no GPU driver or graphics library needs to be installed, resulting in lightweight deployment.

[0043] Furthermore, once the static scene data is determined, the physical server can continuously publish dynamic data, as specifically implemented below: Determine the dynamic pose frame and state data corresponding to the virtual model; generate the dynamic data associated with the scene identifier based on the dynamic pose frame and the state data.

[0044] Specifically, the dynamic pose frames corresponding to the virtual model include the position and rotation of the geometry and camera. A dynamic pose frame can be a transformation matrix containing 3D position and quaternion rotation. State data refers to physical states or simulation control states, used to provide simulation time, frame number, pause state, and robot activation state.

[0045] Based on this, the dynamic pose frames and state data corresponding to the virtual model are determined. The state data is the physical state or simulation control state, used to provide simulation time, frame number, pause state, and robot activation state. Dynamic data associated with scene identifiers is generated based on the dynamic pose frames and state data. Simultaneously, the physical device publishes the dynamic pose frames carrying scene identifiers, and publishes the physical state or simulation control state, continuously generating dynamic poses and physical time.

[0046] Continuing with the previous example, the physics process loads a model containing six bipedal robots and a soccer field, building scene version R1. Subsequently, the host native camera renderer obtains static scene data, dynamic poses, and physical states via physics WebSocket, DDS, shared memory, QUIC, or other publish-subscribe systems. Each robot has a head camera, with a target frame rate of 30Hz.

[0047] In summary, the physical terminal publishes dynamic pose frames carrying scene identifiers, as well as physical states or simulation control states. The physical terminal continuously generates dynamic poses and physical time, separating static scene data from dynamic data and reducing the amount of data transmitted subsequently.

[0048] Step 104: Invoke the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data.

[0049] Specifically, after loading the virtual model, determining the static scene data carrying scene identifiers based on the virtual model, and determining the dynamic data associated with the scene identifiers, the image generation module can be invoked. Based on the image generation unit set, initial image data containing at least two types of data is generated from the static scene data and the dynamic data. Here, the image generation module refers to the camera renderer, the image generation unit set contains at least one image generation unit, and each image generation unit refers to the active camera currently generating the image. The image generation unit set is the set of active cameras currently generating the image. The at least two types of data include color and depth. The initial image data is the data obtained through multi-camera color rendering and depth rendering.

[0050] Based on this, after loading the virtual model, determining the static scene data carrying the scene identifier based on the virtual model, and determining the dynamic data associated with the scene identifier, the image generation unit set is determined, the image generation module is called, and initial image data containing at least two types of data is generated based on the image generation unit set, the static scene data, and the dynamic data.

[0051] Furthermore, an image generation unit set can be constructed based on robot state information and / or camera state information, as specifically implemented as follows: Candidate image generation units are determined based on the robot state table and / or camera enable configuration table; at least one image generation unit is determined from the candidate image generation units based on the robot state information and / or camera state information, and the image generation unit set is constructed based on the at least one image generation unit.

[0052] Specifically, the robot status table records the robot's activation state. Based on the robot status information, it can be determined whether the corresponding camera is an active camera. The camera of the robot in the active state is designated as the active camera, and the active camera is stored as an image generation unit in the image generation unit set. The camera enable configuration table records the camera's start or stop state. Based on the camera status information, it can be determined whether the camera is in the start state. The camera in the start state is designated as an image generation unit and stored in the image generation unit set.

[0053] Based on this, the robot state table records the activation state of each robot, and the camera enable configuration table records the on / off state of each camera. All available candidate image generation units are determined according to the robot state table and / or camera enable configuration table. At least one image generation unit is determined from the candidate image generation units based on the robot state information and / or camera state information, and an image generation unit set is constructed based on this at least one image generation unit. The at least one image generation unit is the camera corresponding to the robot in the active state, or the camera in the on state.

[0054] Continuing with the previous example, the set of active cameras that need to generate images is determined based on the robot's activation status or external camera enable configuration. Cameras whose owners are not active or are explicitly turned off are not included in this capture request. The camera topic subscription status can also be used as a camera enable condition to determine whether a camera should be added to the active camera set. Active cameras can also be selected based on explicit enable configuration, number of subscriptions, priority, or GPU budget.

[0055] In summary, at least one image generation unit is determined from the candidate image generation units based on robot state information and / or camera state information, and an image generation unit set is constructed. Robots in an inactive state automatically turn off all their virtual cameras, thereby saving GPU computing resources.

[0056] Furthermore, after determining the dynamic and static scene data, initial image data can be obtained by executing rendering tasks to perform color rendering and depth rendering respectively, as specifically implemented as follows: Based on the dynamic pose frames in the dynamic data, the image generation units in the image generation unit set are combined to obtain an image rendering task; the image generation module is called to execute the image rendering task to obtain a color dataset and a depth dataset; the color data in the color dataset is stored in the color data region of the combined region, and the depth data in the depth dataset is stored in the depth data region of the combined region to obtain the initial image data.

[0057] Specifically, image generation units in the image generation unit set are combined based on dynamic pose frames in the dynamic data. This can be achieved by organizing multiple active cameras corresponding to the same dynamic pose frame into a capture request. There is at least one image rendering task, which can be executed in batches or sequentially. The color dataset stores the rendered color data, and the depth dataset stores the rendered depth data. The color data region can correspond to a color periphery region, and the depth data region can correspond to a depth periphery region. The combined region can refer to the combined GPU readback buffer.

[0058] Based on this, image generation units in the image generation unit set are combined based on the dynamic pose frames in the dynamic data. Multiple active cameras corresponding to the same dynamic pose frame are organized into capture requests to obtain image rendering tasks. The image generation module is called to execute at least two image rendering tasks sequentially or in batches to obtain color and depth datasets. The color data from the color dataset is stored in the color data region of the combined region, and the depth data from the depth dataset is stored in the depth data region of the combined region to obtain initial image data. This achieves the storage of all color and depth data in different regions of the same combined region, and the reading of camera color and depth results can be completed with a single mapping.

[0059] Continuing with the previous example, multiple active cameras corresponding to the same dynamic pose frame are organized into capture requests, and off-screen color and depth rendering is performed sequentially or in batches. For example... Figure 2 As shown, for each camera, color output (color texture) is written to a color attachment, and depth output is written to a depth attachment. Both attachments are copied to different regions of a combined GPU readback buffer, completing the reading of the camera's color and depth results through a single buffer mapping. Multiple cameras can also be rendered to atlases or texture arrays, reducing per-camera synchronization overhead through batch commands and combined readbacks. Multiple cameras can be rendered sequentially, or atlases, texture arrays, multi-viewport, or multi-view rendering can be used.

[0060] In practice, the camera renderer adds cameras belonging to the active robot to the active camera set based on the robot activation identifier in the unified simulation state. Whenever the simulation time increases by at least approximately 1 / 30 of a second relative to the previous capture time, a capture request is generated using the latest dynamic pose. For each active camera, a JPEG RGB and millimeter depth array is output, along with the camera identifier, frame sequence, and simulation time.

[0061] In summary, by storing all color and depth data in different regions of the same combined region, the camera's color and depth results can be read with a single mapping.

[0062] Furthermore, considering the need to render based on both static and dynamic scene data, during the rendering process, updates to the static scene data may cause inconsistencies between the scene identifiers of the static and dynamic scene data. In such cases, it is necessary to abort the camera renderer's image rendering task and request new static scene data. The specific implementation is as follows: Determine the static scene version information corresponding to the static scene data and the dynamic version information corresponding to the dynamic data; if the static scene version information and the dynamic version information are consistent, execute the step of calling the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data and the dynamic data; if the static scene version information and the dynamic version information are inconsistent, obtain the target static scene data.

[0063] Specifically, the static scene version information corresponding to static scene data refers to the scene identifier carried by the static scene data. The dynamic version information corresponding to dynamic data refers to the scene identifier carried by the dynamic data. The target static scene data refers to new static scene data relative to the static scene data, and the target static scene data needs to be requested again from the physical end.

[0064] Based on this, the static scene version information corresponding to the static scene data and the dynamic version information corresponding to the dynamic data are determined. A consistency comparison is performed between the static and dynamic scene version information. If they match, the static scene data is considered usable and the image generation module can be directly invoked. Based on the image generation unit set, initial image data containing at least two types of data is generated from the static and dynamic scene data. If the static and dynamic scene version information do not match, the static scene data is considered invalid and new static scene data needs to be acquired; that is, the target static scene data needs to be obtained.

[0065] Continuing the previous example, the camera renderer running in the host machine's native environment subscribes to static scene data and dynamic pose frames. Instead of parsing the original robot model file, the camera renderer creates GPU meshes, materials, textures, lights, and camera objects based on static scene snapshots and updates object transformations based on dynamic pose frames. When the version of the static scene data is inconsistent with the version of the dynamic pose, image generation is paused and a new static scene is requested to prevent rendering new scene poses using old resources. When the GPU is unavailable, placeholder images can be generated for transmission link testing, but normal operating mode should output real rendered data.

[0066] In practice, the physics process switches to the new model and publishes version R2. The native renderer (camera renderer) detects that the dynamic pose version is inconsistent with the local R1, stops capturing, and requests the R2 static scene. The camera relay receives the clear cache command and deletes the old metadata and image cache. After loading R2, the camera metadata is republished and capturing resumes. The native renderer can run as a standalone process, a thread within the host application, or a native plugin.

[0067] In summary, by comparing the static scene version information with the dynamic scene version information for consistency, and in the event of inconsistency between the two, new static scene data is obtained to prevent object clipping or positional errors caused by rendering new scene poses using old resources.

[0068] Furthermore, after determining the dynamic data, it is possible to determine whether the next capture time has been reached based on the target frame rate and time information, thus avoiding the generation of duplicate images. The specific implementation is as follows: Determine the target frame rate and the time information in the dynamic data; if it is determined that detection is generated through image data based on the target frame rate and the time information, execute the step of calling the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data and the dynamic data.

[0069] Specifically, the target frame rate refers to the target camera frame rate, and the time information refers to the simulation time of the physical state. A detection representation is generated from the image data to determine when the next capture moment has arrived.

[0070] Based on this, the target frame rate is determined, and time information is extracted from the dynamic data. If, based on the target frame rate and time information, it is determined that detection is achieved through image data generation, indicating that the next capture time has arrived according to the simulation time in the physical state and the target camera frame rate, the image generation module is invoked. Based on the image generation unit set, initial image data containing at least two types of data is generated from static scene data and dynamic data. If the simulation time has not progressed, the current state overlaps with the previous capture state, there is no active camera, or the environmental data is incomplete, the current image generation is skipped, allowing the target camera frame rate to follow the simulation time.

[0071] Continuing with the previous example, the system determines whether the next capture time has arrived based on the target camera's frame rate and the simulation time in the physical state. If the simulation time has not progressed, the current state overlaps with the previous capture state, there is no active camera, or the environmental data is incomplete, then the current image generation is skipped. This ensures that the camera frame rate follows the simulation time, rather than continuing to generate repetitive images during pauses.

[0072] In practice, after the user pauses the physical simulation, the physical state may still be transmitted due to heartbeats, but the simulation time will not increase. The camera scheduling module detects that the simulation time is the same as the previous capture, records the reason for skipping the "repeated simulation time," and does not perform rendering or encoding. After resuming the simulation, images are generated again from the new simulation time, avoiding sending a large number of identical frames to the robot during the pause.

[0073] In summary, based on the target frame rate and time information, if detection is determined to be generated through image data, the image generation module is invoked. Based on the image generation unit set, static scene data and dynamic data, initial image data containing at least two types of data is generated to ensure that the target camera frame rate follows the simulation time and avoids generating duplicate images.

[0074] Step 106: Construct target image data based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data.

[0075] Specifically, after the image generation module is invoked to generate initial image data containing at least two types of data based on the image generation unit set, static scene data, and dynamic data, target image data can be constructed based on the image generation unit set, dynamic data, scene identifiers, and initial image data. The target image data includes encoded image data obtained by encoding the initial image data, and also includes camera metadata determined according to the image generation unit set.

[0076] Based on this, after calling the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, static scene data, and dynamic data, target image data is constructed based on the image generation unit set, dynamic data, scene identifiers, and initial image data. The target image data includes encoded image data obtained by encoding the initial image data, as well as camera metadata. The encoded image data is an RGB-D image frame.

[0077] Furthermore, after determining the initial image data, the initial image data can be encoded to establish camera image frames, as specifically implemented below: The color data and depth data are determined from the initial image data; the color data is converted into image color data, and the depth data is converted into a depth array; image generation unit data is determined according to the image generation unit set, and simulation data is determined according to the dynamic data; and the target image data is generated based on the image generation unit data, the simulation data, the image color data, and the depth array.

[0078] Specifically, converting color data to image color data means encoding the corresponding color pixels in RGB format, such as JPEG, PNG, or other image formats. Converting depth data to a depth array means converting the depth values ​​to a fixed-length array with a preset unit, achieving depth unit conversion, such as a millimeter depth array. Metric floating-point, inverse depth, or point cloud methods can also be used. Image generation unit data refers to camera metadata, while simulation data includes camera identifier, camera name, frame sequence, simulation time, scene version, etc. Simulation data, image color data, and depth arrays constitute a camera image frame.

[0079] Based on this, color and depth data are determined from the initial image data. The color data is encoded into image color data in JPEG, PNG, or other image formats, and the depth data is converted into a depth array. Image generation unit data is determined based on the image generation unit set, camera metadata is constructed, and simulation data is determined based on dynamic data. Camera image frames are generated based on the simulation data, image color data, and depth array, and target image data is constructed based on the camera image frames and camera metadata.

[0080] Following the previous example, color pixels are encoded into JPEG, PNG, or other image formats, and depth values ​​are converted into fixed-length arrays with preset units, such as millimeter depth arrays. Camera identifier, camera name, frame sequence, simulation time, scene version, RGB data, and depth data are encapsulated into camera image frames. RGB data can be in JPEG, PNG, H.264, AV1, or uncompressed format. Camera metadata is encapsulated separately and updated when parameters change. Camera metadata may include: camera identifier and name; image width and height; intrinsic parameter matrix K; distortion parameter D; projection matrix P; correction matrix R; near and far clipping planes; and the robot and coordinate system identifier. Camera image frames may include: message type and payload length; camera identifier and name; frame sequence number; simulation time or timestamp; scene version; compressed RGB data; depth data; and image generation status or error flag.

[0081] In summary, camera image frames are generated based on simulation data, image color data, and depth arrays. Target image data is then constructed based on these camera image frames and camera metadata, thus establishing a unified camera image frame and transforming unstructured data into structured data, facilitating subsequent network transmission or storage.

[0082] Furthermore, after determining the target image data, the data can be published and subscribed through the publishing and subscription entry points provided by the data processing module, as detailed below: The data processing module is invoked to provide a data publishing entry to the image generation module, which is used to publish the target image data, and to provide a data subscription entry to the container module, which is used to subscribe to the target image data; the data processing module is used to send image data to the algorithm module in the container module according to the data sending rules.

[0083] Specifically, the data processing module is the camera relay running within the container. The camera relay can expose a publishing entry point to the host machine (the camera renderer running in the host machine's native environment) and a subscription entry point to subscribers inside the container. Subscribers can be algorithm modules within the container module. The data sending rules can be camera theme sending rules, that is, maintaining the sending status according to the camera theme.

[0084] Based on this, the data processing module, acting as a camera relay running within the container, can call upon the image generation module to provide a data publishing entry point and a data subscription entry point to the container module. The data publishing entry point is used to publish target image data to the image generation module. The data subscription entry point is used to subscribe to target image data to the container module. The data processing module also sends image data to the algorithm module within the container module according to data sending rules. Specifically, it maintains the sending status according to camera themes. The algorithm module within the container module acts as a subscriber; when a camera theme of the same subscriber is sending data, newly arriving frames replace the old frames awaiting transmission for that theme. If the subscriber's sending buffer exceeds a threshold, potentially lossy camera frames are skipped.

[0085] Continuing the previous example, a camera relay runs within the container. The camera relay exposes a publishing entry to the host machine and a subscription entry to the container's internal network. The camera renderer connects to the publishing entry mapped to the host machine to send camera metadata and camera frames; robot RGB, depth, or other perceptual extensions connect to the container's internal subscription entry and subscribe by topic. This topology does not require the container to actively connect to a fixed host domain name, thus ensuring compatibility across different container runtimes. The camera relay can maintain sending status by camera topic. When a camera topic for the same subscriber is being sent, a newly arriving frame replaces the old frame awaiting transmission for that topic; if the subscriber's sending buffer exceeds a threshold, potentially lossy camera frames are skipped. Camera metadata and necessary control messages are reliably sent, and the most recent metadata is cached for use by new subscribers. Figure 3 As shown, the renderer generates F1, the camera relay sends F1 to the slow subscriber, generates F2 and sets it as a frame to be sent, generates F3 to replace the unsent F2, F1 is sent, and F3 is sent directly without the expiration of F2.

[0086] The camera relay's subscription port within the container listens only on the local loopback address, allowing the robot within the container to extend its connection. The publish entry listens on a mappable address and maps it to a random port on the host. The host's native renderer connects to this random port, thus eliminating the need to rely on `host.docker.internal` or the container to access the host gateway. When a new RGB subscriber subscribes to a camera metadata topic, the relay sends the most recent metadata from its cache in camera identifier order. Subsequently, the subscriber subscribes to a specific camera image topic and only receives the corresponding images. The robot debugger takes 100ms to process one frame, while the renderer produces images at 30Hz. The relay only stores one frame to be sent during the transmission of the current frame; new frames overwrite older frames to be sent. Therefore, the images received by subscribers may drop to approximately 10Hz, but each frame is close to the current simulation state, preventing the accumulation of several seconds of delay.

[0087] In summary, camera relays can maintain transmission status according to camera topics, achieving latest frame backpressure for each topic and preventing slow consumers from blocking. Running camera relays within containers exposes a publishing entry point to the host machine and a subscription entry point to the container, simplifying container deployment topology and improving the distribution efficiency and real-time performance of target image data.

[0088] Furthermore, considering that the generation of target image data involves multiple data processing nodes such as data construction, rendering, and encoding, any abnormality in any data processing node may lead to abnormalities in the generation process of target image data. Therefore, it is necessary to record the generation chain of target image data and handle abnormalities in the generation process. The specific implementation is as follows: The data processing information and anomaly information associated with the target image data are determined; the data processing information is stored; and an anomaly handling task is generated based on the anomaly information. The anomaly recovery is completed by executing the anomaly handling task.

[0089] Specifically, the data processing information associated with the target image data refers to the link performance during the target image data generation process, i.e., the execution time and skipping reasons of each data processing node. Data processing nodes include, but are not limited to, static scene snapshot generation, camera metadata construction, initial image data generation, GPU readback, and the encoding, publishing, and relay transmission of initial image data, used for performance segmentation statistics. Anomaly information can include physical connection disconnection and publishing connection disconnection. Anomaly handling tasks can include physical connection reconnection and publishing connection reconnection. Anomaly handling tasks can also be new session initialization tasks, i.e., clearing the old scene state and camera cache when a new session is established to avoid cross-session reuse.

[0090] Based on this, the system identifies data processing information associated with the target image data, including static scene snapshot generation, camera metadata construction, initial image data generation, GPU readback, and the encoding, publishing, and relay transmission of the initial image data. It also identifies anomaly information corresponding to the target image data processing chain and implements performance segmentation statistics. The system stores data processing information and generates anomaly handling tasks based on this information. By executing these tasks, anomaly recovery is achieved, enabling limited backoff and reconnection after physical or published connection loss. Furthermore, when a new session is established, the system clears old scene states and camera caches to prevent cross-session reuse.

[0091] In summary, data processing information is stored, and anomaly handling tasks are generated based on anomaly information. By executing these anomaly handling tasks, anomaly recovery can be achieved, ensuring the efficiency of target image data generation.

[0092] This specification provides a data processing method according to one embodiment, which loads a virtual model, determines static scene data carrying scene identifiers based on the virtual model, and determines dynamic data associated with the scene identifiers. An image generation module is invoked to generate initial image data containing at least two types of data based on an image generation unit set, the static scene data, and the dynamic data. Target image data is constructed based on the image generation unit set, the dynamic data, the scene identifiers, and the initial image data to ensure consistency in scene version among the initial image data output by each image generation unit, avoiding image anomalies caused by version mismatch between static scene and dynamic data, and reducing the burden of repeated transmission or loading of static scene data. Static scene data does not need to be repeatedly transmitted, reducing data retransmission overhead.

[0093] The following is in conjunction with the appendix Figure 4 Taking the application of the data processing method provided in this specification in the generation and transmission of robot simulation images as an example, the data processing method will be further explained. Figure 4 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0094] Step 402: Load the virtual model and determine the robot model and environment model contained in the virtual model.

[0095] Step 404: Extract robot data from the robot model description file of the robot model, and extract scene data from the environment model description file of the environment model.

[0096] Step 406: Construct static scene data based on robot data, scene data, scene identifiers corresponding to the virtual model, and camera parameters.

[0097] Step 408: Determine the dynamic pose frame and state data corresponding to the virtual model, and generate dynamic data associated with scene identifiers based on the dynamic pose frame and state data.

[0098] Step 410: Combine the image generation units in the image generation unit set based on the dynamic pose frames in the dynamic data to obtain the image rendering task.

[0099] Step 412: Call the image generation module to perform the image rendering task and obtain the color dataset and depth dataset.

[0100] Step 414: Store the color data from the color dataset into the color data region of the combined region, and store the depth data from the depth dataset into the depth data region of the combined region to obtain the initial image data.

[0101] Step 416: Determine the color data and depth data in the initial image data, convert the color data into image color data, and convert the depth data into a depth array.

[0102] Step 418: Determine the image generation unit data based on the image generation unit set, and determine the simulation data based on the dynamic data, and generate the target image data based on the image generation unit data, simulation data, image color data, and depth array.

[0103] Step 420: Invoke the data processing module to provide a data publishing entry point to the image generation module. The data publishing entry point is used to publish the target image data. Also, provide a data subscription entry point to the container module. The data subscription entry point is used to subscribe to the target image data.

[0104] Step 422: The data processing module is used to send image data to the algorithm module in the container module according to the data sending rules.

[0105] Step 424: Determine the data processing information and anomaly information associated with the target image data, store the data processing information, generate anomaly handling tasks based on the anomaly information, and complete the anomaly recovery by executing the anomaly handling tasks.

[0106] In practical applications, data processing systems can be robot simulation systems. For example... Figure 5 As shown, during the simulation system's operation, it receives physical state data and visual state data sent by the physical terminal. Before rendering, a pre-verification is performed: checking whether the current scene version is consistent and the data is complete. If the version does not match or the data is incomplete, new static scene data is requested from the physical terminal and a waiting state is entered; if the verification passes, it further determines whether the simulation time has advanced and reached the preset capture period. If the time has not advanced or the capture period has not been reached, the capture is skipped.

[0107] After pre-verification, the set of currently active image generation units is determined. If the set of active units is empty, the current image generation is skipped to avoid invalid rendering; if the set is not empty, the GPU scene transformation is updated according to the latest dynamic pose data, and then off-screen rendering is performed to generate color and depth data corresponding to each active unit. After rendering is complete, all data is read back at once through the combined buffer, the color data is encoded and compressed, and the depth data is converted into a numerical array of preset units. Finally, it is encapsulated into target image data and distributed according to the theme corresponding to each image generation unit. If the same theme has not been sent in the previous frame, the new frame replaces the old frame.

[0108] The data processing system provided in this embodiment eliminates the need for a GPU graphics environment in the container, reducing deployment complexity and improving compatibility with environments such as Docker and Podman. The native camera renderer does not re-parse the original model, reducing inconsistencies between the physical and rendered models. Cameras are scheduled at simulation time, maintaining sensor semantic correctness during pauses and non-real-time runtime. Active cameras are selected based on subscriptions, reducing unnecessary GPU rendering, readback, and encoding. Color and depth share a readback completion path, reducing GPU synchronization and mapping overhead. Subject-specific latest frame replacement limits end-to-end latency and memory usage for slow consumers. Camera metadata caching enables new consumers to quickly obtain complete calibration information. Performance segmentation statistics facilitate the identification of rendering, readback, encoding, or network bottlenecks.

[0109] Corresponding to the above method embodiments, this specification also provides data processing system embodiments. Figure 6 A schematic diagram of the structure of a data processing system according to one embodiment of this specification is shown. Figure 6 As shown, the data processing system 600 includes a physical terminal 610, an image generation module 620, a data processing module 630, and an algorithm module 640. The physical terminal 610 is used to load a virtual model, determine static scene data carrying scene identifiers based on the virtual model, and determine dynamic data associated with the scene identifiers; and send the static scene data and the dynamic data to the image generation module 620. The image generation module 620 is used to generate initial image data containing at least two types of data based on an image generation unit set, the static scene data, and the dynamic data; construct target image data based on the image generation unit set, the dynamic data, the scene identifiers, and the initial image data, and send the target image data to the data processing module 630 through a data publishing entry point. The data processing module 630 is used to receive subscription information for the target image data from the algorithm module and send the target image data to the algorithm module 640.

[0110] The data processing system includes a physical terminal, an image generation module, a data processing module, and an algorithm module. For example... Figure 7As shown, the physical layer loads the virtual model, determines static scene data carrying scene identifiers based on the virtual model, and determines the dynamic data associated with the scene identifiers (including dynamic pose and simulation time). Then, the static scene data and dynamic data are sent to the image generation module. The image generation module runs in the host machine's native environment and generates initial image data containing color and depth data based on the image generation unit set, static scene data, and dynamic data. It then constructs target image data based on the image generation unit set, dynamic data, scene identifiers, and initial image data, and sends the target image data to the data processing module through the container camera publishing entry. The data processing module is deployed inside the container and receives subscription information for the target image data from the algorithm module. It then distributes the target image data to the algorithm module according to the theme corresponding to each image generation unit. The algorithm module includes RGB publishing extensions, depth publishing extensions, and robot perception programs that subscribe to different themes, used to receive the target image data and perform subsequent perception processing.

[0111] This specification provides a data processing system in one embodiment, comprising a physical terminal, an image generation module, a data processing module, and an algorithm module. The physical terminal loads a virtual model, determines static scene data carrying scene identifiers based on the virtual model, and determines dynamic data associated with the scene identifiers. The static scene data and dynamic data are sent to the image generation module. The image generation module generates initial image data containing at least two types of data based on an image generation unit set, the static scene data, and the dynamic data. Target image data is constructed based on the image generation unit set, the dynamic data, the scene identifiers, and the initial image data, and is sent to the data processing module through a data publishing entry. This ensures consistency in scene version among the initial image data output by each image generation unit, avoiding image anomalies caused by version mismatch between static scene and dynamic data, and reducing the burden of repeated transmission or loading of static scene data. Static scene data does not need to be repeatedly transmitted, reducing data retransmission overhead. The data processing module receives subscription information for the target image data from the algorithm module and sends the target image data to the algorithm module.

[0112] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 8 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 8 As shown, the device includes: The loading unit 802 is configured to load a virtual model, determine static scene data carrying scene identifiers based on the virtual model, and determine dynamic data associated with the scene identifiers. The generation unit 804 is configured to call the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data and the dynamic data. Construction unit 806 is configured to construct target image data based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data.

[0113] In an optional embodiment, determining the static scene data carrying scene identifiers based on the virtual model includes: Determine the robot model and environment model included in the virtual model; Extract robot data from the robot model description file of the robot model, and extract scene data from the environment model description file of the environment model; The static scene data is constructed based on the robot data, the scene data, the scene identifier corresponding to the virtual model, and the camera parameters.

[0114] In an optional embodiment, determining the dynamic data associated with the scene identifier includes: Determine the dynamic pose frame and state data corresponding to the virtual model; The dynamic data associated with the scene identifier is generated based on the dynamic pose frame and the state data.

[0115] In an optional embodiment, the construction of the image generation unit set includes: Candidate image generation units are determined based on the robot state table and / or camera enable configuration table; At least one image generation unit is determined from the candidate image generation units based on robot state information and / or camera state information, and the image generation unit set is constructed based on the at least one image generation unit.

[0116] In an optional embodiment, the invocation of the image generation module, based on the image generation unit set, generates initial image data containing at least two types of data from the static scene data and the dynamic data, including: Based on the dynamic pose frames in the dynamic data, the image generation units in the image generation unit set are combined to obtain the image rendering task. The image generation module is invoked to perform the image rendering task, and the color dataset and depth dataset are obtained. The initial image data is obtained by storing the color data from the color dataset into the color data region of the combined region, and storing the depth data from the depth dataset into the depth data region of the combined region.

[0117] In an optional embodiment, constructing target image data based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data includes: The color data and the depth data are determined from the initial image data; The color data is converted into image color data, and the depth data is converted into a depth array; Image generation unit data is determined based on the image generation unit set, and simulation data is determined based on the dynamic data. The target image data is generated based on the image generation unit data, the simulation data, the image color data, and the depth array.

[0118] In an optional embodiment, before invoking the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the method further includes: Determine the static scene version information corresponding to the static scene data, and the dynamic version information corresponding to the dynamic data; If the static scene version information is consistent with the dynamic version information, the step of calling the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data and the dynamic data is executed. If the static scene version information is inconsistent with the dynamic version information, the target static scene data is obtained.

[0119] In an optional embodiment, after determining the dynamic data associated with the scene identifier, the method further includes: Determine the target frame rate and extract time information from the dynamic data; If, based on the target frame rate and the time information, it is determined that detection is generated through image data, the step of calling the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data is executed.

[0120] In an optional embodiment, the data processing method further includes: The data processing module is invoked to provide a data publishing entry to the image generation module, which is used to publish the target image data, and a data subscription entry is provided to the container module, which is used to subscribe to the target image data; The data processing module is used to send image data to the algorithm module in the container module according to the data sending rules.

[0121] In an optional embodiment, the data processing method further includes: Determine the data processing information and anomaly information associated with the target image data; The data processing information is stored, and an exception handling task is generated based on the exception information. The exception handling task is executed to complete the exception recovery.

[0122] One embodiment of this specification provides a data processing apparatus that loads a virtual model, determines static scene data carrying scene identifiers based on the virtual model, and determines dynamic data associated with the scene identifiers. An image generation module is invoked to generate initial image data containing at least two types of data based on an image generation unit set, the static scene data, and the dynamic data. Target image data is constructed based on the image generation unit set, the dynamic data, the scene identifiers, and the initial image data to ensure consistency in scene version among the initial image data output by each image generation unit, avoiding image anomalies caused by version mismatch between static scene and dynamic data, and reducing the burden of repeated transmission or loading of static scene data. Static scene data does not need to be repeatedly transmitted, reducing data retransmission overhead.

[0123] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0124] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0125] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0126] In one embodiment of this specification, the above-described components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0127] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0128] The processor 920 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0129] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0130] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0131] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the data processing method described above.

[0132] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0133] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0134] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0135] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0136] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0137] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0138] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. A data processing method, characterized in that, include: Load a virtual model, determine static scene data carrying scene identifiers based on the virtual model, and determine the dynamic data associated with the scene identifiers; The image generation module is invoked to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data. Target image data is constructed based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data.

2. The data processing method according to claim 1, characterized in that, The step of determining static scene data carrying scene identifiers based on the virtual model includes: Determine the robot model and environment model included in the virtual model; Extract robot data from the robot model description file of the robot model, and extract scene data from the environment model description file of the environment model; The static scene data is constructed based on the robot data, the scene data, the scene identifier corresponding to the virtual model, and the camera parameters.

3. The data processing method according to claim 1, characterized in that, The dynamic data associated with the scene identifier includes: Determine the dynamic pose frame and state data corresponding to the virtual model; The dynamic data associated with the scene identifier is generated based on the dynamic pose frame and the state data.

4. The data processing method according to claim 1, characterized in that, The construction of the image generation unit set includes: Candidate image generation units are determined based on the robot state table and / or camera enable configuration table; At least one image generation unit is determined from the candidate image generation units based on robot state information and / or camera state information, and the image generation unit set is constructed based on the at least one image generation unit.

5. The data processing method according to claim 1, characterized in that, The image generation module, based on the image generation unit set, generates initial image data containing at least two types of data from the static scene data and the dynamic data, including: Based on the dynamic pose frames in the dynamic data, the image generation units in the image generation unit set are combined to obtain the image rendering task. The image generation module is invoked to perform the image rendering task, and the color dataset and depth dataset are obtained. The initial image data is obtained by storing the color data from the color dataset into the color data region of the combined region, and storing the depth data from the depth dataset into the depth data region of the combined region.

6. The data processing method according to claim 5, characterized in that, The construction of target image data based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data includes: The color data and the depth data are determined from the initial image data; The color data is converted into image color data, and the depth data is converted into a depth array; Image generation unit data is determined based on the image generation unit set, and simulation data is determined based on the dynamic data. The target image data is generated based on the image generation unit data, the simulation data, the image color data, and the depth array.

7. The data processing method according to claim 1, characterized in that, Before the image generation module is invoked, and initial image data containing at least two types of data is generated based on the image generation unit set, the method further includes: Determine the static scene version information corresponding to the static scene data, and the dynamic version information corresponding to the dynamic data; If the static scene version information is consistent with the dynamic version information, the step of calling the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data and the dynamic data is executed. If the static scene version information is inconsistent with the dynamic version information, the target static scene data is obtained.

8. The data processing method according to claim 1, characterized in that, After determining the dynamic data associated with the scene identifier, the method further includes: Determine the target frame rate and extract time information from the dynamic data; If, based on the target frame rate and the time information, it is determined that detection is generated through image data, the step of calling the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data is executed.

9. The data processing method according to claim 1, characterized in that, The data processing method further includes: The data processing module is invoked to provide a data publishing entry to the image generation module, which is used to publish the target image data, and a data subscription entry is provided to the container module, which is used to subscribe to the target image data; The data processing module is used to send image data to the algorithm module in the container module according to the data sending rules.

10. The data processing method according to claim 1, characterized in that, The data processing method further includes: Determine the data processing information and anomaly information associated with the target image data; The data processing information is stored, and an exception handling task is generated based on the exception information. The exception handling task is executed to complete the exception recovery.

11. A data processing system, characterized in that, It includes a physical terminal, an image generation module, a data processing module, and an algorithm module, including: The physical terminal is used to load a virtual model, determine static scene data carrying scene identifiers based on the virtual model, and determine dynamic data associated with the scene identifiers; and send the static scene data and the dynamic data to the image generation module. The image generation module is used to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data; construct target image data based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data; and send the target image data to the data processing module through the data publishing entry. The data processing module is used to receive subscription information for the target image data from the algorithm module and send the target image data to the algorithm module.

12. A data processing apparatus, characterized in that, include: The loading unit is configured to load a virtual model, determine static scene data carrying scene identifiers based on the virtual model, and determine dynamic data associated with the scene identifiers. The generation unit is configured to call the image generation module to generate initial image data containing at least two types of data based on the image generation unit set, the static scene data, and the dynamic data. The construction unit is configured to construct target image data based on the image generation unit set, the dynamic data, the scene identifier, and the initial image data.

13. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 10.

15. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 10.

Images