Data processing device and method for converting data

Through the generator and converter mechanism of the data processing device, data structure information is extracted and conversion plug-ins are generated, which solves the problem of efficient conversion between different data structures, reduces computing overhead, and is suitable for embedded systems.

CN118901058APending Publication Date: 2024-11-05ROBERT BOSCH GMBH
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202280093908.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-01-20
Filing Date
2022-12-28
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently convert sensor data or log data between different data structures, especially in embedded systems with limited computing resources, resulting in data processing devices requiring themselves to perform complex format conversions, increasing computing overhead.

Method used

Provide a data processing device, which extracts original data and data structure information through the input interface, the generator selects appropriate source code to generate plug-ins and merges to generate conversion source code, and the converter executes plug-ins to convert data into target format, avoiding manual programming and repeat the conversion process.

Benefits of technology

It realizes flexible conversion of data structures, reduces the computing overhead at the sending end, improves data processing efficiency, and can adapt to changes in various data structures. It is especially suitable for embedded systems with limited computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118901058A_ABST
    Figure CN118901058A_ABST
Patent Text Reader

Abstract

According to various embodiments, a data processing device is described, which has an input interface, which is designed to receive input data and to extract raw data from the input data and information about a data structure according to which the raw data is structured, the data structure comprises a plurality of data types described by the information; for at least one data type used by the data structure, a source code generation plug-in associated with the data type is selected from a plurality of source code generation plug-ins, and a conversion source code for converting the data type into a target format is generated by means of the selected source code generation plug-in. Merging the conversion source code for converting the data type into the target format into the conversion source code for converting the data structure into the target format, and generating a conversion plug-in for converting the data structure into the target format by compiling the conversion source code for converting the data structure into the target format; and having a converter provided for executing the conversion plug-in and thus converting the raw data into a target format.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a data processing apparatus and a method for converting data. Background Art

[0002] The recorded and transmitted sensor data or log data is usually converted into a suitable data format for processing. Data conversion is the migration of recorded data between formats used by different systems in order to benefit from the different advantages of different systems, utilize the data in different fields, and exchange data with external parties.

[0003] For performance reasons, the data can be recorded and transmitted, for example, as raw data, i.e. the output format of the data conversion can be a very compact format that only enables the separation and identification of data blocks in the data stream. The receiver (i.e. the receiving data processing device) must then interpret these data based on a complex data structure, for example for analysis software. For example, in certain applications, such as when sending data from an embedded system, the computing resources in the sender (i.e. the data processing device sending the data) are limited, and the sender only provides the raw data as is, without adapting it to the requirements of the receiver.

[0004] It is desirable for a recipient to be able to flexibly and efficiently process data from different sources structured according to different data structures. Summary of the invention

[0005] According to different embodiments, a data processing device is provided, which has an input interface, which is configured to receive input data and extract raw data and information about a data structure from the input data, wherein the raw data is structured according to the data structure, wherein the data structure contains multiple data types described by the information; a generator, which is configured to select a source code generation plug-in assigned to the data type from multiple source code generation plug-ins for at least one data type used by the data structure, generate a conversion source code for converting the data type into a target format with the help of the selected source code generation plug-in, merge the conversion source code for converting the data type into the target format into a conversion source code for converting the data structure into the target format, and generate a conversion plug-in for converting the data structure into the target format by compiling the conversion source code for converting the data structure into the target format; and a converter, which is configured to execute the conversion plug-in and thereby convert the raw data into the target format.

[0006] The data processing device described above obviously implements a Bootstrap mechanism, which enables it to receive data and convert it into the desired format without preparing manually programmed conversion routines in advance for each data structure according to which the data structure can be structured. Instead, the data processing device constructs the conversion routine of the data structure from the data types that make up the data structure and the source code that exists for the conversion of the data type. This can be done, for example, by recursion in the processing of the data structure. The conversion routine only needs to be created once for each data type. When the data structure used changes rapidly (for example, in the case of updating the software in the vehicle control device), the receiving data device (for example, a host for analyzing data from the vehicle) can also process the raw data. This makes it possible, in particular, to keep the overhead of the sending data processing device low, because it does not have to convert itself in order to provide data in a certain data format (i.e., data structured according to one or more certain data structures).

[0007] Only the source code generation plug-in is retained in the generator, that is, one plug-in for each data type (that should not or cannot be converted one-to-one), instead of the plug-in for all possible data structures. This enables the generator to implement a group of access (data type) conversion rules, and therefore generates the correct conversion program code for one or more data structures used. For the data types used by the data structures that can be converted one-to-one, general conversion routines can be used, that is, for these data types, it is not necessary to require the generation of their own source code plug-ins (but use general conversion source codes). For data types that cannot be converted one-to-one (i.e., generally), a corresponding source code generation plug-in is set. The source code generation plug-in is different for different (these) data types. Therefore, "at least one data type" describes, for example, all data types for which a source code generation plug-in is set or that cannot be converted generally.

[0008] Various embodiments are described below.

[0009] Embodiment 1 is the data processing apparatus described above.

[0010] Embodiment 2 is a data processing device according to embodiment 1, wherein the information on the data structure describes a syntax tree of the data structure.

[0011] This enables efficient description of the data structure used in the information about the data structure (also referred to as meta-information).

[0012] Embodiment 3 is a data processing device according to Embodiment 1 or 2, wherein the input data includes original data structured according to the data structure for each of a plurality of data structures, and the generator is configured to generate a corresponding conversion plug-in for each data structure, and the converter is configured to obtain the extracted original data according to which data structure is structured and execute the corresponding conversion plug-in, thereby converting the original data into a target format.

[0013] Thus, the converter is enabled to correctly convert raw data structured according to the plurality of data structures into a target format.

[0014] Embodiment 4 is a data processing device according to Embodiment 3, wherein the generator is configured to assign a hash value to each conversion plug-in, and wherein the converter is configured to obtain and execute a conversion plug-in for converting original data structured according to one of the data structures by forming a hash value of a syntax tree of the data structure and selecting a conversion plug-in assigned the formed hash value.

[0015] The use of hash values ​​of the syntax tree enables simple generation of unique identifiers and thus simple selection of the correct transformation plugin for the data structure.

[0016] Embodiment 5 is a data processing device according to any one of Embodiments 1 to 4, wherein the original data is measurement data provided by the robot device (for example, read from its memory) and / or log data of program execution on the robot device.

[0017] This enables the robotic device (even an embedded system) to transmit measurement data or log data as raw data without having to keep a certain data format. This reduces the computational overhead on the robotic device side, as computational resources are usually limited.

[0018] Embodiment 6 is a method for converting data, the method comprising: receiving input data; extracting original data and information about a data structure from the input data, the original data being constructed according to the data structure, the data structure containing multiple data types described by the information; for at least one data type used by the data structure, selecting a source code generation plug-in assigned to the data type from multiple source code generation plug-ins, using the selected source code generation plug-in to generate a conversion source code for converting the data type into a target format, merging the conversion source code for converting the data type into a conversion source code for converting the data type into a target format, and generating a conversion plug-in for converting the data structure into the target format by compiling the conversion source code for converting the data structure into the target format, and executing the conversion plug-in, thereby converting the original data into the target format.

[0019] Embodiment 7 is a computer program having instructions which, when executed by one or more processors, cause the one or more processors to perform the method described above.

[0020] Embodiment 8 is a computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the method described above.

[0021] It should be noted that the embodiments described in connection with the data processing apparatus apply analogously to the method for converting data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In the accompanying drawings, similar reference numerals generally refer to the same components in completely different views. The accompanying drawings are not necessarily drawn to scale, but emphasis is generally placed on illustrating the principles of the invention. In the description hereinafter, various aspects are described with reference to the following drawings.

[0023] Figure 1 Show DETAILED DESCRIPTION

[0024] The following detailed description refers to the accompanying drawings, which illustrate specific details and aspects of the present disclosure in which the present invention can be practiced for interpretation. Other aspects may be used and structural, logical and electrical changes may be made without departing from the scope of protection of the present invention. Various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0025] Different examples are described in more detail below.

[0026] Figure 1 A vehicle 101 is shown.

[0027] exist Figure 1 In the example of FIG. 1 , a vehicle 101 , such as a passenger car or a truck, is equipped with a vehicle control device (such as an electronic control unit (ECU)) 102 .

[0028] The vehicle control device 102 has data processing components such as a processor (eg, a CPU (Central Processing Unit)) 103 and a memory 104 for storing control software 107 according to which the vehicle control device 102 operates, and data processed by the processor 103 .

[0029] For example, the stored control software (computer program) has instructions which, when executed by the processor, cause the processor 103 to implement driving assistance functions or also collect data (sensor data or driving data) or even autonomously control the vehicle.

[0030] By collecting data or log data related to the execution of the control software, the vehicle control device 102 generates a certain amount of raw data, which is recorded by the recording device 105. To do this, the vehicle control device 102 only needs to write the data to a common memory (such as memory 104) and transfer ownership to the recording device 105 (i.e., transfer the data to the recording device 105, but do not copy it). The data is then transmitted from the device to the server 108 via the network 106. This can be done continuously or in bundles, for example, when the vehicle 101 is in a garage, it can be transmitted through larger files.

[0031] For example, in the case of a control device in a vehicle, the network 106 is a wireless network. In the case of a control device transmitting data to a static robotic device (eg, an industrial robot arm), the network 106 may be, for example, a wired network (eg, Ethernet).

[0032] The server 108 performs subsequent processing depending on what is recorded for the data. For example, this may include analyzing the execution of the control software or analyzing the operation of the vehicle, in particular by displaying the data for visual perception by the user. Suitable software tools can be used for this purpose, but these tools usually require a certain data format. Therefore, it is required to convert the transmitted raw data into a certain data format (i.e., convert it). This conversion is in turn related to the format of the information contained in the raw data, that is, how the raw data is generated from the corresponding information (such as measured values ​​and log messages from the control software, etc.). Therefore, the server 108 must perform a format conversion. For example, the server 108 executes the following analysis software, which requires ROS (Robot Operating System) data.

[0033] The generation or parameterization of data conversion routines for converting data from a source format into a target format requires information about the corresponding format structure (i.e., layout). For example, the generation of a conversion routine for format conversion (offline in the server 108 or also online in the vehicle 101, for example to display the data on an on-board display) requires knowledge of the source code of the control program 107 that generated the raw data and of the tool used to generate the control program 107. However, this is not given in some cases, for example because it is desirable to develop the measuring system and the system for analyzing the measured data independently of each other and at different times, for example because the conversion routine is only needed when the data are fed to the server 108 (e.g., downloaded from a cloud storage that originally transmitted the data) or when a developer requests data for analysis purposes and his software tool requires a certain format.

[0034] Therefore, according to different embodiments, instead of generating the conversion routine for the data provided by the software itself together with the generation of the software, a method of use is provided: in the case of the conversion routine, the conversion routine is generated by meta-information about the data format (e.g., data structure, such as C++ data structure) used by the data processing device for the original data to be sent. The conversion routine generated by the meta-information of the data structure (e.g., C++ data structure) is automatically generated on the receiving side (e.g., in the server 108). It should be noted that the layout of the original data also depends on the corresponding compiler and the corresponding hardware. Therefore, before the conversion, the layout of the machine performing the conversion must first be translated so that the corresponding conversion routine can play a role. This needs to be generally implemented according to different embodiments (keyword ABI, i.e., memory alignment, padding, byte order, etc.).

[0035] The meta information is delivered with the original data (e.g., in a common data stream). The generated conversion routine can use standard rules to reconstruct the original information in the target format, or use user-defined rules to optimize the conversion (e.g., use specialized target data types for conversion as needed). The meta information can deserialize the original data of the recipient (e.g., server 108) into the target format.

[0036] According to different embodiments, the recipient may generate a conversion routine for self-describing data (raw data plus metadata) without providing information about the generation process of the software that generated the raw data. The conversion routine may be generated (e.g., generated on the fly) based on the data received by the recipient. The prerequisite is that the recipient has its own rules on how to interpret and convert all data types of the raw data. The receiver's converter (i.e., data converter) uses the converter to convert the received raw data into the target data format.

[0037] According to various embodiments, a generator is first provided, which (in the generator phase) generates one or more conversion routines that are then used by the converter (in the conversion phase) to convert received raw data.

[0038] Figure 2 The generator stages performed by generator 201 are shown.

[0039] The generator stage can also be regarded as a bootstrap stage. The generator 201 receives data format information of information contained in the input raw data 203 from the meta information reader 204. The information is attached to the input raw data 204 as meta information (for example, included in the input data stream together with the input raw data 203).

[0040] Generator 201 is a conversion plug-in generator, i.e., generates conversion routines in the form of conversion plug-ins 203. To this end, generator 201 determines the data type for input raw data 203, and how to read and convert input raw data 203. Generator controller 211 receives program code of corresponding mapping rules of determined data types from relevant mapping rule plug-ins in the group of mapping rule plug-ins 205 to which generator 201 belongs. Defining mapping rules using mapping rule plug-ins 205 allows users to extend. Mapping rule plug-ins generate source code, such as executable templates, that converts data types into target formats.

[0041] The generator generates source code for converting data structures to a target format from source code for converting data types using data structures upon which the original data structures are based. The source code generator receives source code 206 for converting data types (e.g., C++ to ROS - multiple data type conversion source code) and additional source code 211 for one-time activities that can be used to convert to a target format, such as initializing data elements (e.g., messages) for the target format.

[0042] Then, the generator control 211 sends the source codes 206 and 211 to the compiler 207, and the compiler generates an executable conversion plug-in 208 (the converter can directly load the plug-in).

[0043] Among them, the mapping plug-in 205 is used to define mapping rules and generate corresponding data type conversion source code, and is used in the generator stage (for example, as Python code with Jinja template); the conversion plug-in 203 is generated by the generator 201 in the generator stage (for example, generated from optimized C++ code to achieve high data sets) and is used in the conversion stage.

[0044] For example, the target data format is a ROS message.

[0045] The determination of one or more data structures for the original data 203 can be regarded as a reflection mechanism. The meta information reader 202 reads meta information from the deserialized input data provided by the input data reader 209 and the data deserializer 210. For example, these units use a deserialization framework, which includes tools and libraries for deserializing data streams containing original C++ objects, with source ABI (i.e., directly detected from memory). It can be deserialized into any data format through a reflection mechanism. With this framework, original C++ data can be processed without serialization on the data processing device that sends it. According to one embodiment, the deserialization framework uses so-called class information and ABI flags to describe the data type (standard) layout, and uses class hash values ​​to identify the data type layout. Independence can be achieved by inserting this information into the record stream so that it can be deserialized in the corresponding log data sink. Here, it is transmitted to the meta information reader 202.

[0046] For example, the meta information reader 202 builds an AST (Abstract Syntax Tree) from the meta information 204 using a deserialization framework and sends it to the generator 201 .

[0047] For example, the conversion plug-in generated by generator 201 provides name mapping of C++ data types and namespaces, and logic for generating copy instructions (such as memcpy instructions) to convert deserialized raw data into corresponding objects (such as ROS message objects) in the target data format.

[0048] For example, as input to compiler 207, generator 201 produces

[0049] Generate code for generating ROS messages from the ROS package corresponding to the data type used in the raw data 211

[0050] C++ transformation code 206 and CMake project for creating transformation plugins

[0051] Auxiliary scripts for simply compiling ROS package code and converting plugin code

[0052] According to an embodiment, the generator 201 generates a transformation plug-in 208 for each data structure (eg, each C++ root type) used in the original data 203, and generates a file that exports the transformation plug-in 208 so that the dynamic loader can find it.

[0053] For example, the generator 201 can be configured so that it can generate a conversion plug-in 208 for any C++ data structure. It can recursively decompose C++ structures (structs) into primitives or data types and generate mapping plug-ins 205 for them. Therefore, the generator 201 can create a routine for converting to the target format for each data type encountered in the recursive process.

[0054] According to different embodiments, since the input data (i.e., the original data 203 and the meta-information 204) is self-contained in offline situations (such as files or file streams) and online situations (socket streams), if (i.e., once or until) the generator 201 (such as a receiver, such as a server that implements the generator 201) receives the input data, the generator 201 can generate a transformation plug-in (or multiple transformation plug-ins).

[0055] The generator 201 provides the conversion plug-in 208 to the converter. Figure 3 As shown, the converter is equipped with a conversion plug-in.

[0056] Figure 3 The conversion stages performed by converter 301 are shown.

[0057] The converter 301 receives raw input data 302 with meta information 303. For example, the converter 301 can be a precompiled executable program that dynamically loads (eg as a shared library) the conversion plug-ins 208, 304 generated by the generator 201 (based on the meta information 204, 303).

[0058] The converter 301 has an input data reader 305 and a data deserializer 306, for example using the deserialization framework 307 described above. The converter 301 also includes an actual conversion routine 309, which uses a conversion plugin 304 and, depending on the target format, can also use a library 308 to generate the target format (such as roscpp).

[0059] For example, to identify the correct transformation plugin 304 for a data structure, the transformation routine 309 uses the class hash value (eg, C++ class hash value) of the data structure, ie, for example, the hash value of the AST of the data structure (eg, class).

[0060] For a group of input data, as long as the AST of the data type does not change, the conversion plug-in 304 only needs to be generated once. Even small changes, such as changes in the name or component or parent class of a class (which should not affect the conversion at first glance), also cause the hash value of the class to change. On the other hand, there are some minor changes that do not change the hash value of the class (because it does not affect the ABI (Application Binary Interface), such as C++ ABI).

[0061] The conversion routine 309 uses the conversion framework formed by the conversion plug-in 304 to generate output data (such as output data stream) 310 in the target format. These data can be directly stored in the process of offline transmission and analysis (such as in the case of ROS, stored in the ROS bag file), or received and processed by other components in the network during online transmission and analysis.

[0062] In summary, according to different embodiments, a data processing device is provided, which can execute the following Figure 4 The method shown.

[0063] Figure 4 A flow chart 400 is shown illustrating a method for converting data according to one embodiment.

[0064] In 401 , input data is received.

[0065] In 402 , raw data and information about a data structure are extracted from the input data, the raw data being structured according to the data structure, wherein the data structure contains a plurality of data types described by the information.

[0066] At 403, for at least one data type used by the data structure,

[0067] In 404, a source code generation plug-in that is matched to the data type is selected from a plurality of source code generation plug-ins. (Each of the plurality of source code generation plug-ins is matched to a corresponding data type);

[0068] In 405, using the selected source code generation plug-in, a conversion source code is generated for converting the data type into a (predetermined or predefined) target format.

[0069] At 406, merging conversion source code that converts the data type into the target format into conversion source code that converts the data structure into the target format; and

[0070] In 407 , a conversion plug-in for converting the data structure into the target format is generated by compiling the conversion source code for converting the data structure into the target format.

[0071] At 408, a conversion plug-in is selected and the raw data is converted into a target format.

[0072] Figure 4The method can be performed by means of a data processing device (e.g., a computer) of one or more data processing units. The term "data processing unit" can be understood as an entity of any type that enables processing of data or signals. The data or signal can be processed, for example, according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit of a programmable gate array (FPGA), or any combination thereof, or may be formed by an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit of a programmable gate array (FPGA), or any combination thereof. Any other way of implementing the corresponding functions described in more detail herein can also be understood as a data processing unit or a logic circuit device. One or more of the method steps described in detail herein can be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0073] For example, Figure 4 The method is used to receive data from a robotic device. The term "robotic device" may be understood to refer to any technical system, such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system.

[0074] Although specific embodiments have been shown and described herein, it is recognized by those skilled in the art that the specific embodiments shown and described may be replaced with different alternatives and / or equivalent implementations without departing from the scope of protection of the present invention. The application should cover any adaptation or variation of the specific embodiments discussed herein. Therefore, the present invention is intended to be limited only by the claims and their equivalents.

Claims

1. A data processing device, comprising: An input interface, the input interface being configured to receive input data and extract raw data and information about a data structure from the input data, the raw data being structured according to the data structure, wherein The data structure includes a plurality of data types described by the information; a generator configured to generate, for at least one data type used by the data structure, Selecting a source code generation plug-in corresponding to the data type from a plurality of source code generation plug-ins; Generate conversion source code for converting the data type into a target format by means of the selected source code generation plug-in; Merging conversion source code for converting data types into a target format into conversion source code for converting a data structure into a target format; and By compiling a conversion source code for converting the data structure into the target format, a conversion plug-in for converting the data structure into the target format is generated; and A converter is arranged to execute the conversion plug-in and thereby convert the raw data into a target format.

2. The data processing device according to claim 1, wherein: Information about the data structure A syntax tree describing the data structure.

3. The data processing device according to claim 1 or 2, wherein: The input data includes original data for each of a plurality of data structures, the original data being structured according to the data structure, and the generator is configured to generate a corresponding conversion plug-in for each data structure, and the converter is configured to obtain, for the extracted original data: according to which data structure the original data is structured, and execute the corresponding conversion plug-in to convert the original data into a target format.

4. The data processing device according to claim 3, wherein: The generator is configured to assign a hash value to each transformation plug-in, and the converter is configured to determine and execute a transformation plug-in for transforming raw data structured according to one of the data structures by forming a hash value of a syntax tree of the data structure and selecting a transformation plug-in assigned the formed hash value.

5. The data processing device according to any one of claims 1 to 4, wherein: The raw data is measurement data provided by the robot device and / or log data of program execution on the robot device.

6. A method for converting data, the method comprising: Receive input data; Extracting raw data and information about a data structure from the input data, the raw data being structured according to the data structure, wherein: The data structure includes a plurality of data types, the data types being described by the information; For at least one data type used by the data structure, Selecting a source code generation plug-in corresponding to the data type from a plurality of source code generation plug-ins; Generate conversion source code for converting the data type into a target format by means of the selected source code generation plug-in; Merging conversion source code for converting the data type into a target format into conversion source code for converting the data structure into a target format; and Generate a conversion plug-in for converting the data structure into the target format by compiling a conversion source code for converting the data structure into the target format; and The conversion plug-in is executed and thereby converts the raw data into a target format.

7. A computer program having instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to claim 6.

8. A computer-readable medium storing instructions, which, when executed by one or more processors, cause the one or more processors to perform the method according to claim 6.

Citation Information

Cited By

  • Data format conversion method and device based on dynamic lazy loading, equipment and medium

    CN119996510A

  • Data format conversion method, device, equipment and medium based on dynamic lazy loading

    CN119996510B