Data exchange method of pipeline element and non-transient computer readable medium

Abstract

A data exchange method of a pipeline element is suitable for processing a pipeline of a data stream, and the method comprises the following steps: processing first original data in the data stream by a general element in the pipeline to generate first analysis data, receiving the first analysis data by a first docking element, and processing the first analysis data to generate first feedback data, and receiving data associated with the first feedback data by using the general element, and processing second original data in the data stream according to the data associated with the first feedback data.

Description

Technical Field

[0001] The invention relates to a data exchange method of pipeline elements and a non-transient computer-readable medium. Background Art

[0002] Using a pipeline-based multimedia architecture to build applications is a common and convenient development method that can build specific functions (such as image acquisition, encoding, and decoding) into a single element. Since elements are swappable and quickly combined, data exchange can be easily achieved.

[0003] However, once the pipeline is executed, it is difficult to obtain the internal information in the middle, resulting in low reusability of streaming data. Furthermore, due to the limitation of the pipeline architecture, if you need to change the relevant parameters during the running process, you need to restart the pipeline. Summary of the invention

[0004] In view of the above, the present invention provides a pipeline element data exchange method and a non-transitory computer-readable medium that solve the above problems.

[0005] A data exchange method for pipeline elements according to an embodiment of the present invention is applicable to a pipeline for processing data streams, the method comprising: processing first original data in the data stream with a universal element in the pipeline to generate first analysis data; receiving the first analysis data with a first docking element, and processing the first analysis data to generate first feedback data; and receiving data associated with the first feedback data with the universal element, and processing second original data in the data stream according to the data associated with the first feedback data.

[0006] According to an embodiment of the present invention, a non-transitory computer-readable medium includes one or more computer-executable instructions. After a computer loads the one or more computer-executable instructions, the computer executes the above-mentioned pipeline element data exchange method.

[0007] In summary, the data exchange method of pipeline elements and the non-transitory computer-readable medium according to the above embodiments of the present invention can enable the elements in the pipeline to dynamically exchange data, and the reusability of the stream data is high. By docking elements, other data operations can be performed synchronously, and another pipeline can also be executed in the docking element, so that the application of the pipeline is more extensive.

[0008] The above description of the content of the present invention and the following description of the embodiments are used to demonstrate and explain the spirit and principle of the present invention, and to provide further explanation of the claims of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1It is a schematic diagram of pipeline elements and a data exchange method thereof according to an embodiment of the present invention.

[0010] Figure 2 FIG. 4 is a flow chart of a method for exchanging data of pipeline elements according to an embodiment of the present invention.

[0011] Figure 3 The flowchart is a diagram showing how the docking element processes data according to multiple analysis results generated by the common element.

[0012] Figure 4 FIG. 4 is a detailed structural diagram of a pipeline element according to an embodiment of the present invention.

[0013] Figure 5 Based on Figure 4 An example diagram of the operation of pipeline elements is shown.

[0014] Figure 6 It is a schematic diagram of a pipeline element and a data exchange method thereof according to another embodiment of the present invention.

[0015] Figure 7 It is a schematic diagram of a pipeline element and a data exchange method thereof according to another embodiment of the present invention.

[0016] Figure 8 It is a detailed structural diagram of a pipeline element according to another embodiment of the present invention.

[0017] Figure 9(a) to Figure 9(h) To illustrate the Figure 7 An example diagram of the pipeline elements used for image analysis and camera control operations. DETAILED DESCRIPTION

[0018] The detailed features and advantages of the present invention are described in detail in the following embodiments, and the contents are sufficient to enable any person skilled in the art to understand the technical content of the present invention and implement it accordingly, and according to the contents disclosed in this specification, claims and drawings, any person skilled in the art can easily understand the relevant purposes and advantages of the present invention. The following examples further illustrate the viewpoints of the present invention in detail, but do not limit the scope of the present invention in any viewpoint.

[0019] It should be noted that the pipeline elements described herein can be used in a multimedia framework based on a pipeline (e.g., GStreamer), and the elements can be modules with functions such as image acquisition, encoding, decoding, image processing, algorithm execution, inference, and data storage. The data transmission method between pipeline elements can be through a probe callback function, an event callback function of a system memory image, or other communication or inter-process communication (IPC) protocols. In addition, the data types processed by pipeline elements may include artificial intelligence inference data, data logs and monitoring, scheduling guard results, and pipeline control status. The scheduling guard may be accessing each element on the pipeline line to determine whether the status and information of the element meet expectations, or the scheduling guard may detect the status and information of the universal element to which it is connected through one or more of the following docking elements. Accordingly, the stability and correctness of the pipeline during operation can be ensured. In addition, scheduling may include the cycle of executing the above-mentioned access, and the scheduling method may be defined by the user, which is not limited by the present invention.

[0020] Please refer to the figure, Figure 1 FIG. 1 is a schematic diagram of a pipeline element and a data exchange method thereof according to an embodiment of the present invention. Figure 1 In the pipeline, a generic element 10 is included, and a first docking element 11 is connected across both ends of the generic element 10. The generic element 10 and the first docking element 11 can be executed by the same processor or by different processors, and the present invention is not limited thereto. The processor described herein is, for example, a central processing unit, a graphics processor, a microcontroller, a programmable logic controller, or other processors with computing functions.

[0021] The general element 10 is used to process the data in the data stream and input the processed data to the first docking element 11. The first docking element 11 processes the processed data and feeds the processing result back to the general element 10 so that the general element 10 can process the next data in the data stream accordingly. The source of the data stream can be a data stream generating device or an upstream element, etc., and the original data can be at least one data in the data stream or data output by the upstream element. For example, the data stream generating device is a camera, the data stream is a continuous image sequence, and the original data is an image frame.

[0022] For more detailed explanation of the above, please refer to Figure 1 and Figure 2 ,in Figure 2The flowchart of the data exchange method of pipeline elements according to one embodiment of the present invention is shown. The data exchange method of pipeline elements is applicable to a pipeline for processing data streams, and the method comprises: step S101: processing raw data in the data stream with a common element in the pipeline to generate first analysis data; step S103: receiving the first analysis data with a first docking element, and processing the first analysis data to generate first feedback data; and step S105: receiving data associated with the first feedback data with the common element, and processing another raw data in the data stream according to the data associated with the first feedback data.

[0023] In step S101, the universal element 10 processes the first raw data in the data stream to generate first analysis data. The universal element 10 may pre-store a first algorithm, and the universal element 10 processes the first raw data according to the first algorithm. For example, the data stream is a continuous image sequence, the raw data is an image frame, and the first algorithm is an object tracking algorithm. The first analysis data may include the result of the object tracking algorithm.

[0024] In step S103, the first docking element 11 receives the first analysis data from the universal element 10, and processes the first analysis data to generate first feedback data. The first analysis data may include the result of processing the first analysis data. The first docking element 11 may pre-store a second algorithm, and the first docking element 11 processes the first analysis data according to the second algorithm. For example, the first analysis data is the result of object tracking, and the second algorithm is an optical flow algorithm. One implementation of step S103 may be that the first docking element 11 receives a user instruction, and processes the first analysis data according to the user instruction to generate the first feedback data. Taking object tracking as an example, the user instruction may be an external trajectory prediction result.

[0025] In step S105, the universal element 10 receives data associated with the first feedback data from the first docking element 11, and processes the second raw data in the data stream according to the data associated with the first feedback data. The implementation of step S105 may include the universal element 10 directly receiving the first feedback data from the first docking element 11. Continuing with the examples of the object tracking algorithm and the optical flow algorithm, the universal element 10 may further analyze the motion status of the object in the second raw data according to the first feedback data, such as speed, direction and / or acceleration. Moreover, the second analysis data generated by the universal element 10 after processing the second raw data may be input to the first docking element 11, the user device (e.g., a computer, a smart phone, etc.), the data storage device and / or the next universal element. In addition, the implementation of step S105 may also include the universal element 10 receiving data associated with the first feedback data from another element, and the data is generated by the other element processing the first feedback data.

[0026] By using the docking elements to establish a data loopback, data in the pipeline can be easily reused without increasing the complexity of the pipeline, making the operation of the multimedia framework and artificial intelligence more efficient.

[0027] Please refer to Figure 1 and Figure 3 ,in Figure 3 The flowchart is a diagram showing how the docking element processes data according to multiple analysis results generated by the common element. Figure 3 Can be considered as Figure 2 The detailed flow chart of an embodiment of step S103 of the present invention comprises: step S201: storing the first original data and the second original data by the first docking element; and processing the first original data and the second original data by the first docking element to generate the first feedback data.

[0028] In step S201, the first docking element 11 stores the first original data and the second original data, and the generation time of the second original data is later than the generation time of the first original data. In other words, after processing the first original data, the universal element 10 can output the first original data to the first docking element 11. In step S203, the first docking element 11 processes the first original data and the second original data to generate first feedback data.

[0029] Taking the aforementioned optical flow algorithm as an example, the first original data and the second original data are two consecutive image frames. The first docking element 11 executes the optical flow algorithm on the first original data and the second original data to generate a displacement vector of the object, and outputs the displacement vector as the first feedback data to the universal element 10.

[0030] Please refer to Figure 4 ,in Figure 4 FIG. 1 is a detailed structural diagram of a pipeline element according to an embodiment of the present invention. Figure 4 As shown, the universal element 10 may include a buffer 101 and an operator 102, and the buffer 101 is connected to the operator 102. Similarly, the first docking element 11 may include a buffer 111 and an operator 112, and the buffer 111 is connected to the operator 112, wherein the operator 112 may be connected to an application presenting a user interface to obtain the aforementioned user instructions. The buffer may be used to store data, and the operator may be used to pre-store algorithms, process data stored in the buffer, and control other elements. Furthermore, the operator may be a docking handler or a logic controller, which is not limited by the present invention.

[0031] In addition, the element data exchange method may further include defining a sink pad 10a and a source pad 10b in the universal element 10; defining a source pad 11a and a sink pad 11b in the first docking element 11; and connecting the sink pad 10a of the universal element 10 to the source pad 11a of the first docking element 11, and connecting the source pad 10b of the universal element 10 to the sink pad 11b of the first docking element 11. In addition, the universal element 10 may also be provided with a sink pad 10c and a source pad 10d. The sink pad is used to receive data, and the source pad is used to output data. The operator, buffer, sink pad and source pad of each element may be increased or decreased according to usage requirements. Figure 4 The numbers of operators, buffers, sinks, and sources of each element shown in FIG. 1 are merely examples and are not intended to limit the present invention.

[0032] If an element is connected to a docking element, the buffer and operator of the element can be connected to their respective slots and sources, such as Figure 4 If an element is not connected to a docking element, the buffer and operator of the element can be connected in series, with the buffer connected to the slot end and the operator connected to the source end, such as Figure 4 The first docking element 11 in.

[0033] To explain in more detail Figure 4 Please refer to the operation of the pipeline elements of Figure 4 and Figure 5 ,in Figure 5 Based on Figure 4 The operation example diagram of the pipeline elements is shown. It should be noted that Figure 5 The pipeline architecture and Figure 4 The pipeline architecture is similar to Figure 5 The universal element 10' includes two operators, namely a first operator 102 and a second operator 103. The first operator 102 can be used to process raw data, and the second operator 103 can be used to determine which data needs to be transmitted to the first docking element 11. Figure 5 In the image processing, the data stream is a continuous image sequence and the original data is an image frame.

[0034] The universal element 10' receives the first original data D1 through the slot 10c, and the buffer 101 stores the first original data D1. The first operator 102 performs an object tracking algorithm on the first original data D1 to generate metadata (e.g., an object frame). The second operator 103 uses the metadata as the first analysis data D2, and outputs the first analysis data D2 to the first docking element 11 through the source 10b. The second operator 103 can also output the first original data D1 to the first docking element 11 through the source 10b. The first docking element 11 processes the first analysis data D2 to generate first feedback data, and outputs the first feedback data to the slot 10a of the universal element 10. At the same time, the source 10d of the universal element 10' can output another original data D3 to its downstream element, wherein the original data D3 is data that has been processed by the universal element 10' and the first docking element 11. In addition, after obtaining the first feedback data, the source 10d of the universal element 10' can output the metadata and / or the first original data D1 to its downstream element.

[0035] Please refer to Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a pipeline element and a data exchange method thereof according to another embodiment of the present invention. Figure 6 In the pipeline, the first universal element 21 and the second universal element 22 are included, and the first docking element 11 is connected across the first universal element 21 and the second universal element 22. The first docking element 11, the first universal element 21 and the second universal element 22 can be executed by the same processor or by different processors, which is not limited by the present invention. The first universal element 21 is connected to the second universal element 22. The implementation of the first universal element 21 and the second universal element 22 can be the same as Figure 1 The first docking element 11 can be the same as the universal element 10 of Figure 1 The first docking element 11 is the same.

[0036] The first docking element 11 and the second universal element 22 are suitable for Figures 2 to 5 how it works, and Figure 2 The implementation of step S105 may include the second universal element 22 processing the second original data according to the intermediate data from the first universal element 21, wherein the intermediate data is generated by the first universal element 21 processing the first feedback data. In other words, after the first docking element 11 receives the first analysis data from the second universal element 22 and outputs the first feedback data to the first universal element 21, the first universal element 21 processes the first feedback data to generate the intermediate data, and the second universal element 22 processes the second original data according to the intermediate data.

[0037] For example, the first universal element 21 is used to execute an artificial intelligence inference engine, the second universal element 22 is used to execute an algorithm, and the first docking element 11 is a module connected to the cloud. The first docking element 11 executes inference to generate an inference result (intermediate data), and the second universal element 22 further executes an algorithm on the inference result to generate a calculation result (first analysis data). If the first docking element 11 determines that the inference engine of the first docking element 11 is not suitable after processing the calculation result, an engine replacement instruction is generated and / or a new inference engine (first feedback data) is obtained, so that the first universal element 21 can update the inference engine accordingly.

[0038] In addition, Figure 4 The first docking element 11, the first universal element 21 and the second universal element 22 can each be provided with an operator, a buffer, a slot end and a source end, and the number can be adjusted according to the use requirements. The source end of the first docking element 11 can be connected to the slot end of any upstream element, and can transfer data to the upstream element and perform specified tasks on the upstream element. The pipeline element data exchange method according to the present invention can coordinate the cooperation of different elements, optimize the data processing flow, and according to Figure 6 The architecture enables upstream elements to perform specific tasks according to demand, improving the efficiency of data processing and transmission.

[0039] Figure 7 FIG. 1 is a schematic diagram of a pipeline element and a data exchange method thereof according to another embodiment of the present invention. Figure 7 In the pipeline, the first universal element 21 and the second universal element 22 are included, the first docking element 11 is connected to both ends of the second universal element 22, and the second docking element 12 is connected to both ends of the first universal element 21 and the second universal element 22. The first docking element 11, the second docking element 12, the first universal element 21 and the second universal element 22 can be executed by the same processor or by different processors, which is not limited by the present invention. The first universal element 21 is connected to the second universal element 22. The implementation methods of the first docking element 11, the first universal element 21 and the second universal element 22 can be respectively the same as Figure 6 The first docking element 11, the first universal element 21 and the second universal element 22 are the same.

[0040] The second docking element 12 may receive the first analysis data from the second general element 22, and process the first analysis data to generate second feedback data. The second docking element 12 may output the second feedback data to the first general element 21, or may output the second feedback data to another pipeline or external device.

[0041] Please refer to Figure 7 and Figure 8 ,in Figure 8The detailed structure diagram of the pipeline element according to another embodiment of the present invention is shown. The data exchange method of the pipeline element of the present invention may include defining a source end 11a and a slot end 11b in the first docking element 11; defining a source end 12a and a slot end 12b in the second docking element 12; defining slot ends 21a and 21c and source ends 21b and 21d in the first general element 21; defining slot ends 22a and 22c and source ends 22b and 22d in the second general element 22; connecting the slot end 21a of the first general element 21 to the source end 12a of the second docking element 12; connecting the source end 21d of the first general element 21 to the slot end 22c of the second general element 22; connecting the slot end 22a of the second general element 22 to the source end 11a of the first docking element 11; and connecting the source end 22b of the second general element 22 to the slot end 11b of the first docking element 11 and the slot end 12b of the second docking element 12.

[0042] The second universal element 22 receives the first original data from the first universal element 21, and the first universal element 21 receives the second feedback data from the second docking element 12, and Figure 2 The implementation of step S105 may include the second general element 22 processing the second original data according to the intermediate data from the first general element 21 , wherein the intermediate data is generated by the first general element 21 processing the second feedback data.

[0043] To explain in more detail Figure 7 and Figure 8 For details on how the pipeline elements work, please refer to Figure 9(a) to Figure 9(h) ,in Figure 9(a) to Figure 9(h) To illustrate the Figure 7 A sample diagram of the pipeline elements used for image analysis and camera control operations, and Figure 9(a) to Figure 9(h) Corresponding to the first time point to the eighth time point respectively. Figure 9(a) to Figure 9(h) In , the data stream is a continuous image sequence, and the original data is an image frame. Figure 9(a) to Figure 9(h) The first universal element 21 is used to control the camera, the second universal element 22 is used to execute the object tracking algorithm, the first docking element 11 is used to execute the optical flow algorithm, and the second docking element 12 is used to calculate the control parameters of the camera.

[0044] At the first time point of FIG9(a), the first universal element 21 receives the first original data f(1). The first universal element 21 has not received the control parameter of the second docking element 12, does not process the first original data f(1), and transmits the first original data f(1) to the second universal element 22.

[0045] At the second time point of FIG. 9( b ), the first universal element 21 receives the second original data f(2). The first universal element 21 has not yet received the control parameters of the second docking element 12, and does not process the second original data f(2), and transmits the second original data f(2) to the second universal element 22. The first original data f(1) arrives at the second universal element 22. Since the second universal element 22 has not yet received the result of the optical flow algorithm of the first docking element 11, it does not process the first original data f(1), and transmits the first original data f(1) to the first docking element 11 and the downstream element 23 for subsequent processing.

[0046] At the third time point of FIG. 9( c ), the first universal element 21 receives the third raw data f(3). The first universal element 21 has not yet received the control parameters of the second docking element 12, and does not process the third raw data f(3), and transmits the third raw data f(3) to the second universal element 22. The second raw data f(2) arrives at the second universal element 22. Since the second universal element 22 has not yet received the result of the optical flow algorithm of the first docking element 11, it does not process the second raw data f(2), and transmits the second raw data f(2) to the first docking element 11 and the downstream element 23 for subsequent processing.

[0047] At the fourth time point of FIG. 9( d ), the first universal element 21 receives the fourth raw data f(4). The first universal element 21 has not yet received the control parameters of the second docking element 12 and does not process the fourth raw data f(4), and transmits the fourth raw data f(4) to the second universal element 22. The third raw data f(3) arrives at the second universal element 22. Since the second universal element 22 has not yet received the result of the optical flow algorithm of the first docking element 11, it does not process the third raw data f(3). The second universal element 22 transmits the third raw data f(3) to the first docking element 11 and the downstream element 23 for subsequent processing. At this time, the first docking element 11 has the first raw data f(1) and the second raw data f(2) stored in the buffer.

[0048] At the fifth time point in FIG. 9( e ), the first universal element 21 receives the fifth raw data f(5). The first universal element 21 has not received the control parameters of the second docking element 12 and does not process the fifth raw data f(5), and transmits the fifth raw data f(5) to the second universal element 22. The fourth raw data f(4) arrives at the second universal element 22. Since the second universal element 22 has not received the result of the optical flow algorithm of the first docking element 11, it does not process the fourth raw data f(4). The second universal element 22 transmits the fourth raw data f(4) to the first docking element 11 and the downstream element 23 for subsequent processing. At this time, the buffer of the first docking element 11 stores the first raw data f(1), the second raw data f(2) and the third raw data f(3). The first docking element 11 uses the first raw data f(1) and the second raw data f(2) as references to calculate the displacement vector of the optical flow in the current image to obtain the motion trajectory of each feature point, and transmits the displacement vector to the second universal element 22. After completing the operation of the displacement vector, the first docking element 11 may remove the first original data f(1) from the buffer.

[0049] At the sixth time point of FIG. 9( f ), the first universal element 21 receives the sixth raw data f(6). The first universal element 21 has not received the control parameters of the second docking element 12 and does not process the sixth raw data f(6), and transmits the sixth raw data f(6) to the second universal element 22. The second universal element 22 receives the result of the optical flow algorithm of the first docking element 11 and does not process the fifth raw data f(5). The second universal element 22 transmits the fifth raw data f(5) to the first docking element 11 and the downstream element 23 for subsequent processing. At this time, the buffer of the first docking element 11 stores the second raw data f(2), the third raw data f(3) and the fourth raw data f(4). The first docking element 11 uses the second raw data f(2) and the third raw data f(3) as references to calculate the displacement vector of the optical flow in the current image to obtain the motion trajectory of each feature point, and transmits the displacement vector to the second universal element 22. The second universal element 22 can analyze the motion direction of the object in one step and output the motion direction to the second docking element 12. After completing the operation of the displacement vector, the first docking element 11 may remove the second original data f(2) from the buffer.

[0050] At the seventh time point in FIG. 9( g ), the first general element 21 receives the seventh raw data f(7). The first general element 21 has not received the control parameters of the second docking element 12 and does not process the seventh raw data f(7), and transmits the seventh raw data f(7) to the second general element 22. The second general element 22 receives the result of the optical flow algorithm of the first docking element 11 and does not process the sixth raw data f(6). The second general element 22 transmits the sixth raw data f(6) to the first docking element 11 and the downstream element 23 for subsequent processing. At this time, the buffer of the first docking element 11 stores the third raw data f(3), the fourth raw data f(4) and the fifth raw data f(5). The first docking element 11 uses the third raw data f(3) and the fourth raw data f(4) as references to calculate the displacement vector of the optical flow in the current image to obtain the motion trajectory of each feature point, and transmits the displacement vector to the second general element 22. The second general element 22 can analyze the motion direction of the object in one step and output the motion direction to the second docking element 12. The second docking element 12 determines that the shooting angle of the camera needs to be adjusted according to the movement direction, and generates corresponding control parameters. The control parameters may include at least one of a pan parameter, a tilt parameter, a zoom parameter, and an analog control. After completing the calculation of the displacement vector, the first docking element 11 may remove the third raw data f(3) from the buffer.

[0051] At the eighth time point in FIG. 9(h), the first general element 21 receives the eighth raw data f(8). Since the first general element 21 receives the control parameters of the second docking element 12, the first general element 21 controls the shooting angle of the camera according to the control parameters. The first general element 21 does not process the eighth raw data f(8), and transmits the eighth raw data f(8) to the second general element 22. The second general element 22 receives the result of the optical flow algorithm of the first docking element 11, and does not process the seventh raw data f(7). The second general element 22 transmits the seventh raw data f(7) to the first docking element 11 and the downstream element 23 for subsequent processing. At this time, the buffer of the first docking element 11 stores the fourth raw data f(4), the fifth raw data f(5) and the sixth raw data f(6). The first docking element 11 uses the fourth raw data f(4) and the fifth raw data f(5) as references to calculate the displacement vector of the optical flow in the current image to obtain the motion trajectory of each feature point, and transmits the displacement vector to the second general element 22. The second general element 22 can analyze the moving direction of the object in one step and output the moving direction to the second docking element 12. After completing the calculation of the displacement vector, the first docking element 11 can remove the fourth original data f(4) from the buffer.

[0052] In addition, Figure 7 , Figure 8 and Figure 9(a) to Figure 9(h) In the embodiment, the configuration of the first universal element 21 can also be omitted, so that the second universal element 22 directly receives the data stream, and the second feedback data generated by the second docking element 12 can be output to another pipeline or external device (for example, a control element of a camera) via the source end 12a.

[0053] All or part of the steps in the method described in the above embodiment of the present invention can be implemented by a computer program, such as any combination of an application program, a driver, an operating system, etc. A person skilled in the art can write the method of the above embodiment of the present invention into a computer program code as one or more computer executable instructions and store it in a non-transitory computer readable medium, which will not be described for simplicity. The computer program product implemented according to the method of the above embodiment of the present invention can be stored in an appropriate non-transitory computer readable medium, such as a DVD, a CD-ROM, a flash drive, a hard disk, or a network server that can be accessed through a network (e.g., the Internet, or other appropriate media). In one embodiment, a non-transitory computer readable medium stores a computer program product of the above embodiment, and the computer program product executes a data exchange method for pipeline elements when loaded by a data processing device such as a computer.

[0054] In summary, the data exchange method of pipeline elements and the non-transient computer-readable medium according to the above-mentioned embodiment of the present invention can enable the elements in the pipeline to dynamically exchange data, the streaming data has high reusability, and the relevant parameters can be changed online without restarting the pipeline. Through the docking element, other data operations can be performed synchronously, and another pipeline can be executed in the docking element, so that the application of the pipeline is more extensive. In addition, data can be directly exchanged or controlled between elements in the pipeline, or between elements in the pipeline and external elements. Therefore, the docking element can transmit the data in the pipeline to the outside, which can improve the transparency of the data. In addition, since a large amount of data may be stored in the log or debug system of the multimedia framework based on the pipeline, by reading the data of a certain element separately with the docking element, the generated information can be simpler and more concise, and it can be easier to determine whether the pipeline is abnormal.

[0055]

Explanation of symbols

[0056] 10,10': Universal elements

[0057] 21: First Universal Element

[0058] 22: Second Universal Element

[0059] 11: First docking element

[0060] 12: Second docking element

[0061] 23: Downstream elements

[0062] D1, D3: original data

[0063] D2: First analysis data

[0064] 10b,10d,11a,12a,21b,21d,22b,22d: source

[0065] 10a, 10c, 11b, 12b, 21a, 21c, 22a, 22c: slot end

[0066] 101,111: Buffer

[0067] 102,103,112: Computing Unit

[0068] f(1)~f(8): original data

[0069] S101, S103, S105, S201, S203: steps.

Claims

1. A method for exchanging data of pipeline elements, applicable to a pipeline for processing data streams, characterized in that: The method includes: Processing first raw data in the data stream with the common elements in the pipeline to generate first analyzed data; Receiving the first analysis data with a first docking element, and processing the first analysis data to generate first feedback data; as well as The data associated with the first feedback data is received by the common element, and the second original data in the data stream is processed according to the data associated with the first feedback data.

2. The data exchange method of pipeline elements according to claim 1, characterized in that: Receiving the data associated with the first feedback data with the general element includes: The first feedback data is directly received from the first docking element by the universal element.

3. The element data exchange method according to claim 1, characterized in that: The universal element is connected to a second universal element of the first universal element, and the universal element receives the data associated with the first feedback data, and processes the second original data in the data stream according to the data associated with the first feedback data, including: The second general element processes the second original data according to the intermediate data from the first general element, wherein the intermediate data is generated by the first general element processing the first feedback data.

4. The data exchange method of pipeline elements according to claim 1, characterized in that: Also includes: The first analysis data is received by a second docking element, and the first analysis data is processed to generate second feedback data.

5. The data exchange method of pipeline elements according to claim 4, characterized in that: The universal element is connected to a second universal element of the first universal element, the first universal element receives the second feedback data from the second docking element, the second universal element receives the first original data from the first universal element, the second universal element receives the data associated with the first feedback data, and processes the second original data in the data stream according to the data associated with the first feedback data, including: The second general element processes the second original data according to the intermediate data from the first general element, wherein the intermediate data is generated by the first general element processing the second feedback data.

6. The data exchange method of pipeline elements according to claim 1, characterized in that: Also includes: Defining a sink end and a source end on the universal element and the first docking element respectively; and The slot end of the universal element is connected to the source end of the first docking element, and the source end of the universal element is connected to the slot end of the first docking element.

7. The method for exchanging data of elements according to claim 4, characterized in that: Also includes: defining a sink end and a source end on the universal element, the first docking element, and the second docking element, respectively; and The slot end of the universal element is connected to the source end of the first docking element, and the source end of the universal element is connected to the slot end of the first docking element and the slot end of the second docking element.

8. The data exchange method of pipeline elements according to claim 5, characterized in that: Also includes: defining a sink end and a source end on the first universal element, the second universal element, the first docking element, and the second docking element, respectively; Connecting the slot end of the first universal element to the source end of the second docking element; Connecting the slot end of the second universal element to the source end of the first docking element; as well as The source end of the second universal element is connected to the slot end of the first docking element and the slot end of the second docking element.

9. The data exchange method of pipeline elements according to claim 1, characterized in that: Receiving the first analysis data with the first docking element, and processing the first analysis data to generate the first feedback data comprises: storing the first original data and the second original data in the first docking element; and The first docking element processes the first original data and the second original data to generate the first feedback data.

10. The data exchange method of pipeline elements according to claim 1, characterized in that: Receiving the first analysis data with the first docking element, and processing the first analysis data to generate the first feedback data comprises: Receiving a user instruction with the first docking element; and The first analysis data is processed according to the user instruction to generate the first feedback data.

11. A non-transitory computer-readable medium, characterized in that The method comprises one or more computer executable instructions, and after a computer loads the one or more computer executable instructions, the computer can execute the data exchange method for pipeline elements according to any one of claims 1 to 10.

Images