Data transmission method, device, storage medium and product based on multimedia framework

Abstract

The present application provides a data transmission method, device, storage medium and product based on a multimedia framework. The method includes: using a data source component in the multimedia framework GStreamer to obtain a data stream and a corresponding timestamp; using the data source component to configure the timestamp as a global variable and passing the data stream to at least one processing component according to a preset pipeline, so as to use at least one processing component to process the data stream; if the processing component passes the data stream to the data transmission component according to a preset pipeline, the data transmission component is used to obtain the timestamp configured as a global variable and store the timestamp in a storage space; if the receiving component obtains the data stream transmitted by the data transmission component, the receiving component is used to obtain the timestamp from the storage space. Adding a data transmission component that can obtain the timestamp configured as a global variable can enable the receiving component to obtain the data stream and the corresponding timestamp transmitted by the data transmission component.

Description

Technical Field

[0001] The present application relates to the field of streaming media data transmission, and in particular to a data transmission method, device, storage medium and product based on a multimedia framework. Background Art

[0002] When the sensor captures data, it will generate a timestamp of the current time. Third-party devices can use this timestamp to ensure the consistency of the data captured by each sensor.

[0003] At present, GStreamer is usually used to transmit data streams and timestamps. Specifically, the processing components in GStreamer are used to process the data streams collected by sensors, and the processed data streams and corresponding timestamps are sent to third-party devices through the receiving components. However, some processing components in GStreamer will reorganize the memory, resulting in some customized data such as timestamps being lost during the processing, making it impossible for the receiving components in GStreamer to obtain the timestamps. Summary of the invention

[0004] The present application provides a data transmission method, device, storage medium and product based on a multimedia framework, which are used to solve the problem that some processing components in GStreamer will reorganize the memory, resulting in some customized data such as timestamps being lost during the processing process, making the receiving component in GStreamer unable to obtain the timestamp.

[0005] In a first aspect, the present application provides a data transmission method based on a multimedia framework, comprising: using a data source component in a multimedia framework GStreamer to obtain a data stream and a corresponding timestamp; the timestamp is a timestamp generated according to the time when the data stream is collected; the data stream includes video data, image data, or audio data;

[0006] The data source component is used to configure the timestamp as a global variable and the data stream is passed to at least one processing component according to a preset pipeline, so that the data stream is processed by the at least one processing component;

[0007] If the processing component transmits the data stream to the data transmission component according to the preset pipeline, the data transmission component is used to obtain the timestamp configured as a global variable and store the timestamp in the storage space;

[0008] If the receiving component obtains the data stream transmitted by the data transmission component, the receiving component is used to obtain the timestamp from the storage space.

[0009] In a second aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0010] The memory stores computer-executable instructions;

[0011] The processor executes the computer-executable instructions stored in the memory to implement the data transmission method based on the multimedia framework described in any one of the above aspects.

[0012] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the data transmission method based on the multimedia framework described in any of the above aspects.

[0013] In a fourth aspect, the present application provides a computer program product, comprising computer execution instructions, which, when executed by a processor, implement the data transmission method based on the multimedia framework described in any of the above aspects.

[0014] The data transmission method, device, storage medium and product based on multimedia framework provided by the present application adopt the data source component in multimedia framework GStreamer to obtain data stream and corresponding timestamp; the timestamp is a timestamp generated according to the time when the data stream is collected; the data stream includes video data, image data or audio data; the data source component is used to configure the timestamp as a global variable and pass the data stream to at least one processing component according to a preset pipeline, so as to use at least one processing component to process the data stream; if the processing component passes the data stream to the data transmission component according to a preset pipeline, the data transmission component is used to obtain the timestamp configured as a global variable and store the timestamp in a storage space; if the receiving component obtains the data stream transmitted by the data transmission component, the receiving component is used to obtain the timestamp from the storage space. By configuring the timestamp as a global variable, the transmission mode of the timestamp can be changed, so that the timestamp does not pass through the processing component during the transmission process; adding a data transmission component that can obtain the timestamp configured as a global variable can enable the receiving component to obtain the data stream and the corresponding timestamp transmitted by the data transmission component. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] Figure 1 It is a schematic diagram of an existing data stream processing method based on a multimedia framework;

[0017] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0018] Figure 3A flowchart of a data transmission method based on a multimedia framework provided in Example 1 of the present application;

[0019] Figure 4 A schematic diagram of data transmission provided in an embodiment of the present application;

[0020] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Example 4 of the present application.

[0021] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0023] The terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the following embodiments, "multiple" means more than two, unless otherwise clearly and specifically defined.

[0024] First, the prior art involved in the present invention is described and analyzed in detail.

[0025] The autonomous driving industry requires sensors such as cameras, lidar, millimeter-wave radar, and ultrasonic radar to collect enough data for driving decisions. Due to the instantaneous changes in the surrounding environment, cars need to obtain data from each sensor in real time in order to make timely decisions on changes in the surrounding environment. Each sensor generates a current timestamp when capturing data, and third-party devices can use this timestamp to ensure the consistency of the data captured by each sensor. Therefore, when using GStreamer to pass the data stream from the sensor to the third-party device, the corresponding timestamp is also passed to the third-party device, so that the third-party device can make more accurate decisions based on the data stream and the corresponding timestamp.

[0026] Currently, data source components are usually used to obtain data streams, and multiple GStreamer processing components are used to implement processing functions such as encoding, decoding, rendering, and filtering. Finally, the receiving component sink receives the data stream and saves it or publishes the data stream to a third party through ros. However, some processing components in GStreamer will reorganize the memory, causing some customized data such as timestamps to be lost during the processing process, making it impossible for the receiving component sink in GStreamer to obtain the timestamp.

[0027] Figure 1 It is a schematic diagram of an existing data stream processing method based on a multimedia framework, such as Figure 1 As shown in the figure, the data source component source passes the acquired data stream and the corresponding timestamp to the format conversion component nvvidconv; after the format conversion component processes the data stream, it passes the data stream and the corresponding timestamp to the encoding component omxh265enc; after the encoding component processes the data stream, it passes the data stream to the receiving component sink. The receiving component sink can publish the data stream to a third-party device through the robot operating system ROS. However, the encoding component omxh265enc will reorganize the GStreamer memory, resulting in the loss of custom data such as timestamps during the data stream transmission process, making the timestamp unable to be passed to the receiving component sink.

[0028] The inventor found in the research that the data stream and the timestamp can be transmitted separately, so as to avoid the loss of the timestamp, so that the timestamp does not pass through the processing component during the transmission process, and the receiving component can obtain the data stream and the corresponding timestamp. Specifically, a data transmission component can be added before the receiving component, and the timestamp can be transmitted to the data transmission component by configuring the timestamp as a global variable. Therefore, the present application provides a data transmission method based on a multimedia framework, which uses the data source component in the multimedia framework GStreamer to obtain the data stream and the corresponding timestamp; the timestamp is a timestamp generated according to the time when the data stream is collected; the data stream includes video data, or image data, or audio data; the data source component is used to configure the timestamp as a global variable and pass the data stream to at least one processing component according to a pre-set pipeline, so as to use at least one processing component to process the data stream; if the processing component passes the data stream to the data transmission component according to the pre-set pipeline, the data transmission component is used to obtain the timestamp configured as a global variable and store the timestamp in the storage space; if the receiving component obtains the data stream transmitted by the data transmission component, the receiving component is used to obtain the timestamp from the storage space.

[0029] Figure 2 The following is a schematic diagram of an application scenario provided by an embodiment of the present application, such as Figure 2As shown, it includes: sensor 1, electronic device 2, and third-party device 3. Among them, sensor 1 can be a camera, and electronic device 2 includes multimedia framework GStreamer, and multimedia framework GStreamer includes data source component, at least one processing component, data transmission component, and receiving component. After obtaining the data stream collected by sensor 1 and the corresponding timestamp generated, electronic device 2 executes the data transmission method based on multimedia framework provided by the present application; after completing the processing of the data stream, electronic device 2 can pass the data stream and the corresponding timestamp to third-party device 3.

[0030] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0031] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0032] Embodiment 1

[0033] Figure 3 The data transmission method flow chart based on multimedia framework provided by the first embodiment of the present application, the embodiment of the present application is aimed at the problem that some self-defined data such as timestamp will be lost in the processing process, so that the receiving component sink in GStreamer cannot obtain the timestamp, and provides a data transmission method based on multimedia framework. The method in this embodiment is applied to a data transmission device, and the data transmission device can be located in an electronic device. Among them, the electronic device can be a digital computer representing various forms. Such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers.

[0034] like Figure 3 As shown, the specific steps of this method are as follows:

[0035] Step S101: using the data source component in the multimedia framework GStreamer to obtain a data stream and a corresponding timestamp.

[0036] The timestamp is a timestamp generated according to the time when the data stream is collected; the data stream includes video data, image data, or audio data. The data stream also includes any type of data that can be processed by the multimedia framework GStreamer.

[0037] In the embodiment of the present application, the data source component (full name in English: source element) is the producer of data, provides the data source by acquiring the data stream and the corresponding timestamp and is responsible for the processing of the data source.

[0038] It should be understood that a component (in English: element) can be encapsulated into a plug-in, and a plug-in can contain at least one component. The functional modules in the multimedia framework GStreamer are implemented by pluggable plug-ins, and the plug-ins can be linked to each other and assembled into a pipeline (in English, also known as a pipeline).

[0039] The present application does not limit the method of obtaining the data stream and the corresponding timestamp. For example, the data stream and the corresponding timestamp can be obtained from a storage space, or the input data stream and the corresponding timestamp can be obtained, etc. The specific method can be determined according to actual needs.

[0040] It should be understood that the data transmission method based on the multimedia framework provided in the embodiment of the present application can transmit timestamps and can also transmit custom data corresponding to any data stream other than timestamps. The embodiment of the present application does not limit the transmitted data.

[0041] Step S102: Use the data source component to configure the timestamp as a global variable and pass the data stream to at least one processing component according to a preset pipeline, so as to use the at least one processing component to process the data stream.

[0042] In the embodiment of the present application, the user can pre-arrange and combine components that implement different functions according to specific needs, thereby pre-setting the pipeline. The embodiment of the present application does not limit the method of pre-setting the pipeline. For example, the GStreamer pipeline can be built using any computer language such as c++, python, or cmd, and can also be pre-set using a visual interface.

[0043] In the embodiment of the present application, the order of the components in the pre-set pipeline is: data source component, at least one processing component, data transmission component, receiving component. The embodiment of the present application does not limit the order of multiple processing components in the pre-set pipeline, which can be set according to actual needs.

[0044] In the embodiment of the present application, the processing component can be any one of filters, converters, demuxers, and muxers. The processing component can receive the data stream transmitted by the previous component in the pipeline, operate on the received data stream, and then transmit the data stream to the next component in the pipeline.

[0045] Step S103: If the processing component transfers the data stream to the data transmission component according to the preset pipeline, the data transmission component is used to obtain the timestamp configured as a global variable and store the timestamp in the storage space.

[0046] In an embodiment of the present application, after at least one processing component completes processing of a data stream, a data transmission component can receive the data stream; after receiving the data stream, the data transmission component obtains a timestamp corresponding to the data stream configured as a global variable and stores the timestamp in a storage space.

[0047] Step S104: If the receiving component obtains the data stream transmitted by the data transmission component, the receiving component is used to obtain the timestamp from the storage space.

[0048] Among them, the receiving component (full name in English: sink element) is the consumer of data, which can be responsible for outputting data to a specified device or destination, or saving the received data stream into a file.

[0049] In an embodiment of the present application, after the data transmission component stores the timestamp, the data transmission component passes the data stream to the receiving component; after receiving the data stream, the receiving component obtains the corresponding timestamp from the storage space, and can associate the data stream with the corresponding timestamp for storage or send it to a specified device or destination.

[0050] The method provided in the embodiment of the present application adopts the data source component in the multimedia framework GStreamer to obtain the data stream and the corresponding timestamp; the timestamp is a timestamp generated according to the time when the data stream is collected; the data stream includes video data, or image data, or audio data; the data source component is used to configure the timestamp as a global variable and pass the data stream to at least one processing component according to a preset pipeline, so that the data stream is processed by at least one processing component; if the processing component passes the data stream to the data transmission component according to the preset pipeline, the data transmission component is used to obtain the timestamp configured as a global variable and store the timestamp in a storage space; if the receiving component obtains the data stream transmitted by the data transmission component, the receiving component is used to obtain the timestamp from the storage space. By configuring the timestamp as a global variable, the transmission method of the timestamp can be changed, so that the timestamp does not pass through the processing component during the transmission process; adding a data transmission component that can obtain the timestamp configured as a global variable can enable the receiving component to obtain the data stream and the corresponding timestamp transmitted by the data transmission component.

[0051] Optionally, after step S104, using the receiving component to obtain the timestamp from the storage space, it also includes: the receiving component sends the received data stream and the obtained timestamp to the robot operating system ROS, so as to use the publishing function of ROS to publish the processed data stream and the corresponding timestamp to a third-party device.

[0052] Among them, if the data stream is collected by a sensor installed on the car, the third-party device can be a driving decision-making device, and the driving decision-making device can make decisions on driving behavior based on the received data stream.

[0053] Optionally, the third-party device may also be a playback device to play the video stream processed by the processing component.

[0054] The data transmission method based on the multimedia framework provided by the present application, the receiving component sends the received data stream and the obtained timestamp to the robot operating system ROS, so as to use the publishing function of ROS to publish the processed data stream and the corresponding timestamp to the third-party device, so that the third-party device can ensure the timing consistency of each sensor data stream according to the corresponding timestamp, so as to make more accurate decisions based on the data stream, or ensure the timing consistency of each sensor data stream played.

[0055] Embodiment 2

[0056] Based on the above embodiment, this embodiment involves a refinement of an implementation method of obtaining a data stream and a corresponding timestamp using a data source component in a multimedia framework GStreamer in step S101, specifically including: using a data source component to read a data stream in a memory; using a data source component to read a timestamp in a register.

[0057] The timestamp is generated by the sensor according to the time when the data is captured and stored in the register; the data stream is stored in the memory after the data is captured by the sensor.

[0058] Specifically, after capturing data, the sensor generates a timestamp according to the time when the data is captured and stores the timestamp in a register; the data source component can obtain the data stream by reading the memory DDR, and obtain the timestamp corresponding to the data stream by reading the register.

[0059] The data transmission method based on the multimedia framework provided in the present application adopts a data source component to read the data stream in the memory; adopts a data source component to read the timestamp in the register, that is, adopts a register to temporarily store the timestamp, which can save memory space and improve the processing speed of the timestamp.

[0060] Based on the above embodiment, this embodiment further relates to a refinement of an implementation method of configuring a timestamp as a global variable so that a data transmission component can obtain the timestamp, specifically including:

[0061] Step S102A: After obtaining the timestamp and the data stream, the data source component configures the timestamp as a global variable to write into the dynamic link library.

[0062] In the embodiment of the present application, the dynamic link library stores the global variables corresponding to the data source component, and the timestamp can be written into the dynamic link library by configuring the timestamp as a global variable.

[0063] Step S103A: using the data transmission component to obtain a timestamp configured as a global variable from a dynamic link library.

[0064] Among them, the data transmission component and the data source component share the same dynamic link library.

[0065] In the embodiment of the present application, the data transmission component and the data source component share the same dynamic link library. Since the data transmission component and the data source component share the same dynamic link library, the data transmission component can obtain the timestamp configured as a global variable from the dynamic link library.

[0066] Specifically, the method to make the data transmission component and the data source component share the same dynamic link library is as follows: the interpretation files of the data transmission component and the data source component can be compiled together to achieve the data transmission component and the data source component sharing the same dynamic link library; the data transmission component and the data source component can be separated by naming the data transmission component and the data source component respectively, and the data transmission component and the data source component can be added to the pipeline respectively.

[0067] The data transmission method based on multimedia framework provided by the present application, after the data source component obtains the timestamp and the data stream, configures the timestamp as a global variable to write into the dynamic link library; the data transmission component is used to obtain the timestamp configured as the global variable from the dynamic link library, and the data transmission component and the data source component share the same dynamic link library, so that the data transmission component can obtain the timestamp configured as the global variable, and realize the transmission of the timestamp from the data source component to the data transmission component.

[0068] The following is a description of the data transmission method based on the multimedia framework provided by the present application with reference to a specific example. In this example, the data stream is a video stream, and the processing components include a format conversion component such as nvvidconv and an encoding component such as omxh265enc. Figure 4 A schematic diagram of data transmission provided in an embodiment of the present application, such as Figure 4 As shown, the data transmission method includes the following steps:

[0069] Step 1: Use the data source component to obtain the data stream and the corresponding timestamp and configure the timestamp as a global variable.

[0070] Step 2: After the data source component configures the timestamp as a global variable, the data stream is sent to the format conversion component to perform size cropping and / or format conversion on the data in the data stream.

[0071] For example, nvvidconv is used to crop the image size in the data stream to 640x480, and the image format is converted to NV12.

[0072] Step 3: After the format conversion component completes processing of the data stream, the data stream that has completed format conversion is passed to the encoding component to encode the data in the data stream.

[0073] For example, omxh265enc is used to perform h265 encoding on the pictures in the data stream.

[0074] Step 4: After the encoding component completes processing of the data stream, the encoded data stream is passed to the data transmission component.

[0075] Step 5: The data transmission component obtains the timestamp configured as a global variable and stores the timestamp in a storage space.

[0076] Step 6: The data transmission component passes the data stream to the receiving component, and the receiving component obtains the timestamp corresponding to the data stream from the storage space.

[0077] Embodiment 3

[0078] Based on the above embodiment, this embodiment involves a refinement of the implementation method of storing the timestamp in the storage space in step S103.

[0079] In an embodiment of the present application, an optional way to store the timestamp in the storage space is to store the timestamp in the storage space using an opaque data mechanism.

[0080] In the embodiment of the present application, the opaque data mechanism of GstMiniObject can be used to save the timestamp in the memory of GStreamer to ensure that the timestamp can be transmitted between the data source component and the data transmission component.

[0081] Specifically, the data transmission component can store the sensor timestamp in the GstMiniObject through the gst_mini_object_set_qdata API. When the receiving component obtains the timestamp from the storage space in step S208, the timestamp in the GstMiniObject can be obtained through the gst_mini_object_get_qdata API. Among them, GstMiniObject is a structure for storing data blocks.

[0082] The data transmission method based on the multimedia framework provided in the present application utilizes an opaque data mechanism to store the timestamp in the storage space so that the timestamp can be saved in the memory of GStreamer, thereby ensuring that the timestamp can be transmitted between the data source component and the data transmission component.

[0083] In the embodiment of the present application, another optional way to store the timestamp in the storage space is: using a custom data structure to store the timestamp in the storage space.

[0084] In the embodiment of the present application, the storage of timestamps can be realized by using a registration function to customize the data structure. The customized data structure includes metadata. Metadata can be defined by registering the function GstMeta to pass the metadata to downstream components.

[0085] Specifically, the timestamp may be determined as metadata, and the metadata may be stored in a storage space according to a custom data structure. The embodiment of the present application does not limit the specific custom data structure, and may be defined according to actual needs.

[0086] The data transmission method based on the multimedia framework provided in the present application utilizes a custom data structure to store the timestamp in a storage space, and the timestamp can be saved in the storage space to ensure that the timestamp can be transmitted between the data source component and the data transmission component.

[0087] Optionally, a method of storing the timestamp in the storage space using a custom data structure specifically includes:

[0088] Step S301: register the initialization method of the timestamp in a registration function, wherein the registration function may be GstMeta.

[0089] Step S302: The timestamp is transferred into the buffer, so as to store the timestamp in the buffer in a user-defined data structure according to the initialization method of the timestamp.

[0090] In an embodiment of the present application, the timestamp may be determined as metadata, and the timestamp may be stored in a buffer according to an initialization method corresponding to the metadata.

[0091] Specifically, after using the data transmission component to obtain the timestamp configured as a global variable, the timestamp-related initialization method is registered in the registration function GstMeta by calling the gst_meta_register API, and the sensor timestamp is passed to the buffer buffer by calling the gst_buffer_add_meta API to store the timestamp in the buffer according to the custom data structure.

[0092] The data transmission method based on the multimedia framework provided in the present application registers the initialization method of the timestamp into the registration function and passes the timestamp into the buffer, so that the timestamp can be stored in the buffer according to the custom data structure, so that the receiving component can obtain the timestamp from the storage space.

[0093] Optionally, to ensure that the timestamp stored by the data transmission component and the timestamp obtained by the receiving component are the same timestamp, a corresponding storage type tag needs to be stored while storing the timestamp so that the receiving component can obtain the timestamp according to the storage type, specifically including: storing the type tag of the timestamp in the buffer in a custom data structure according to the timestamp initialization method. The type tag is used to describe the type of the metadata of the timestamp.

[0094] In the embodiment of the present application, the storage of the type tag of the timestamp can be implemented according to a custom data structure. The custom data structure can also include a type tag, so the type tag of the timestamp can be stored in the buffer according to the custom data structure according to the initialization method of the timestamp. The specific storage method of the type tag is similar to the storage method of the timestamp, and will not be repeated here.

[0095] The multimedia framework-based data transmission method provided in the present application stores the type mark of the timestamp in the buffer according to the customized data structure according to the timestamp initialization method, which can ensure that the timestamp stored by the data transmission component and the timestamp obtained by the receiving component are the same timestamp.

[0096] Optionally, based on the above embodiment, in order to enable the receiving component to obtain the timestamp according to the storage type, the following steps are also included: registering the transmission mode of the timestamp in the registration function.

[0097] Specifically, after using the data transmission component to obtain the timestamp configured as a global variable, the timestamp-related initialization method is registered in the registration function GstMeta by calling the gst_meta_register API, and the timestamp transmission method is registered in the registration function. Among them, the initialization method includes a method for storing the timestamp and the corresponding type tag. The timestamp and the corresponding type tag can be initialized according to the initialization method registered in the registration function.

[0098] Optionally, the inverse initialization method of the timestamp may be registered in the registration function GstMeta. According to the inverse initialization method, the storage space occupied by the timestamp may be released.

[0099] When the receiving component is used to obtain the timestamp from the storage space in step S104, the timestamp is obtained from the buffer according to the type mark of the timestamp and the transmission mode of the timestamp.

[0100] Specifically, the timestamp stored in the buffer by the data transmission component can be obtained by the gst_buffer_get_meta API according to the timestamp transmission mode. To ensure the correctness of data acquisition, the type tag described in the above initialization method needs to be used when calling the gst_buffer_get_meta API.

[0101] The data transmission method based on the multimedia framework provided in the present application registers the transmission mode of the timestamp into the registration function; obtains the timestamp from the buffer according to the transmission mode of the timestamp according to the type mark of the timestamp, and can obtain the timestamp from the storage space according to the type mark using the receiving component to ensure that the timestamp stored by the data transmission component and the timestamp obtained by the receiving component are the same timestamp.

[0102] Embodiment 4

[0103] Figure 5 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present application, such as Figure 5 As shown, the present application also provides an electronic device 50 , including: a memory 501 and a processor 502 .

[0104] The memory 501 is used to store computer-executable instructions. Specifically, the program may include program code, and the program code includes computer-executable instructions. The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0105] The processor 502 is used to execute the computer-executable instructions stored in the memory 501 .

[0106] The computer executable instructions are stored in the memory 501 and are configured to be executed by the processor 502 to implement the method provided in any embodiment of the present application. The relevant description can be understood by referring to the relevant descriptions and effects corresponding to the steps in the accompanying drawings, and no further details are given here.

[0107] In the embodiment of the present application, the memory 501 and the processor 502 are connected via a bus. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0108] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are executed by a processor to implement a method provided by any embodiment of the present application.

[0109] An embodiment of the present application also provides a computer program product, including computer execution instructions, which implement the method provided by any embodiment of the present application when executed by a processor.

[0110] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0111] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0112] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0113] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable full-path trajectory fusion device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0116] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0117] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data transmission method based on a multimedia framework, characterized in that: include: The data source component in the multimedia framework GStreamer is used to obtain the data stream and the corresponding timestamp; The timestamp is a timestamp generated according to the time when the data stream is collected; the data stream includes video data, image data, or audio data; The data source component is used to configure the timestamp as a global variable and the data stream is passed to at least one processing component according to a preset pipeline, so that the data stream is processed by the at least one processing component; If the processing component transmits the data stream to the data transmission component according to the preset pipeline, the data transmission component is used to obtain the timestamp configured as a global variable and store the timestamp in the storage space; If the receiving component obtains the data stream transmitted by the data transmission component, the receiving component is used to obtain the timestamp from the storage space.

2. The method according to claim 1, characterized in that The method of using the data source component in the multimedia framework GStreamer to obtain the data stream and the corresponding timestamp includes: Using a data source component to read a data stream in a memory; the data stream is stored in the memory after the sensor captures the data; The data source component is used to read the timestamp in the register; the timestamp is generated by the sensor according to the time when the data is captured and stored in the register.

3. The method according to claim 1, characterized in that The method of using a data source component to configure a timestamp as a global variable includes: After obtaining the timestamp and data stream, the data source component configures the timestamp as a global variable to be written into the dynamic link library; The adopting the data transmission component to obtain the timestamp configured as a global variable includes: A data transmission component is used to obtain a timestamp configured as a global variable from a dynamic link library; the data transmission component and the data source component share the same dynamic link library.

4. The method according to claim 3, characterized in that The storing the timestamp in a storage space includes: The timestamp is stored in a storage space using an opaque data mechanism.

5. The method according to claim 3, characterized in that: The storing the timestamp in a storage space includes: The timestamp is stored in a storage space using a custom data structure.

6. The method according to claim 5, characterized in that The storing the timestamp in a storage space by using a custom data structure includes: Register the timestamp initialization method in the registration function; The timestamp is passed into the buffer, so that the timestamp is stored in the buffer in a customized data structure according to the initialization method of the timestamp.

7. The method according to claim 6, characterized in that Also includes: The type tag of the timestamp is stored in the buffer in a custom data structure according to the initialization method of the timestamp.

8. The method according to claim 7, characterized in that Also includes: Register the timestamp transmission method in the registration function; The adopting a receiving component to obtain the timestamp from a storage space includes: The timestamp is obtained from the buffer according to the type mark of the timestamp and the transmission mode of the timestamp.

9. The method according to any one of claims 1 to 8, characterized in that: After the receiving component is used to obtain the timestamp from the storage space, the method further includes: The receiving component sends the received data stream and the acquired timestamp to the robot operating system ROS, so as to use the publishing function of ROS to publish the processed data stream and the corresponding timestamp to the third-party device.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

12. A computer program product comprising computer executable instructions, characterized in that: When the computer-executable instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images