Data fusion synchronous output circuit and data fusion synchronous output method

Abstract

A preferred embodiment of this application relates to a data fusion synchronous output circuit and a data fusion synchronous output method. The data fusion synchronous output method includes: triggering an analog-to-digital converter to output digital data according to a receiving clock; temporarily storing a first pixel data, a second pixel data, a third pixel data, and the digital data to a buffer synchronous storage circuit according to the receiving clock; performing a specified operation on the first pixel data, the second pixel data, the third pixel data, and the digital data according to the data requirements of a convolutional neural network; and fusing the first pixel data, the second pixel data, the third pixel data, and the digital data after the specified operation according to the data requirements of the convolutional neural network to output data in a specified format.

Description

Technical Field

[0001] This application relates to a data processing technique applicable to neural networks, and in particular to a data fusion synchronous output circuit and a data fusion synchronous output method. Background Technology

[0002] Convolutional Neural Networks (CNNs) are deep learning models widely used in image recognition and processing. Their basic data format is typically homogeneous, meaning they are trained and inferred using image data from the red, green, and blue (R, G, B) channels. However, with technological advancements, researchers have begun exploring the integration of heterogeneous data into CNN models to improve their performance.

[0003] Heterogeneous convolutional neural network models refer to adding other types of data, such as analog-to-digital conversion (ADC) data, to the traditional three-channel image data, forming a four-channel (R, G, B, ADC) structure. The main challenge of this model is ensuring the synchronization and real-time nature of the data. If the camera capture interface controller (CCAP) and the analog-to-digital converter collect data at different sampling times, it is difficult to guarantee the temporal consistency of these data, which will negatively impact the training and inference of the model.

[0004] The choice of data arrangement format is also an important issue during the training of convolutional neural networks. Common arrangement formats include NCHW (batch size, number of channels, height, width) and NHWC (batch size, height, width, number of channels). These two formats differ in computational efficiency, and both increase computational costs when using a processor (CPU) or Direct Memory Access (DMA) to rearrange the data. Summary of the Invention

[0005] Embodiments of this application provide a data fusion synchronous output circuit and a data fusion synchronous output method for integrating data from different sources and performing computation and data fusion according to the needs of a convolutional neural network (CNN).

[0006] A preferred embodiment of this application provides a data fusion synchronous output circuit suitable for organizing heterogeneous data and synchronously outputting it to a convolutional neural network. The heterogeneous data output circuit includes a camera image capture interface control circuit and an analog-to-digital converter. The camera image capture interface control circuit outputs a first pixel data, a second pixel data, and a third pixel data. The analog-to-digital converter converts an external analog signal into digital data. The data fusion synchronous output circuit includes a control circuit, a buffer synchronous storage circuit, a data processing selection unit, and a data synthesis unit.

[0007] The input port of the control circuit is coupled to the camera image capture interface control circuit to receive at least one synchronization signal and one pixel clock signal. The trigger control port of the control circuit is coupled to the analog-to-digital converter. Based on the synchronization signal and the pixel clock signal, the trigger control port of the control circuit outputs a trigger signal to trigger the analog-to-digital converter to perform analog-to-digital conversion and output digital data. The buffer synchronization storage circuit is coupled to the camera image capture interface control circuit and the analog-to-digital converter to receive the first pixel data, the second pixel data, the third pixel data, and the digital data according to a receiving clock. The data operation selection unit is coupled to the control port of the control circuit and the buffer synchronization storage circuit. The data operation selection unit, according to the control of the control circuit, determines the data received from the buffer synchronization storage circuit to perform a specified operation and outputs specified operation data. The data synthesis unit is coupled to the control port of the control circuit and the data operation selection unit. It sequentially receives the specified operation data, arranges the received multiple specified operation data in a specified order according to the control of the control circuit, and outputs the data to the convolutional neural network.

[0008] Another preferred embodiment of this application provides a data fusion synchronous output method, which includes: triggering an analog-to-digital converter to output digital data according to a receiving clock; temporarily storing a first pixel data, a second pixel data, a third pixel data, and the digital data to a buffer synchronous storage circuit according to the receiving clock; performing a specified operation on the first pixel data, the second pixel data, the third pixel data, and the digital data according to the data requirements of a convolutional neural network; and performing data fusion on the first pixel data, the second pixel data, the third pixel data, and the digital data after the specified operation according to the data requirements of the convolutional neural network, and outputting data in a specified format.

[0009] In summary, a preferred embodiment of this application provides a data fusion and synchronous output method that effectively integrates data from different sources and performs computation and data fusion according to the requirements of a convolutional neural network (CNN). By triggering the analog-to-digital converter to output digital data according to the receiving clock and temporarily storing multiple pixel data and digital data in a buffer synchronous storage circuit, the real-time performance and synchronization of the data are ensured. Furthermore, data can be flexibly processed and output according to different data arrangement formats (such as NCHW or NHWC), improving the efficiency and flexibility of the system. During training or inference, normalization and quantization operations are performed on the first, second, and third pixel data and digital data, which helps to improve model performance.

[0010] To further understand the techniques, methods, and effects of this application, reference can be made to the following detailed description and accompanying drawings, which will provide a thorough and concrete understanding of the purpose, features, and concepts of this application. However, the following detailed description and accompanying drawings are for reference and illustration only and are not intended to limit this application. Attached Figure Description

[0011] The accompanying drawings are provided to enable those skilled in the art to further understand this application and are incorporated in and constitute a part of this application's specification. The drawings illustrate exemplary embodiments of this application and are used, together with the description of this application, to explain the principles of this application.

[0012] Figure 1 The diagram shows a circuit block diagram of a data fusion synchronization output circuit according to a preferred embodiment of this application.

[0013] Figure 2 The diagram illustrates the NCHW output result of the data synthesis unit 104 of the data fusion synchronous output circuit according to a preferred embodiment of this application.

[0014] Figure 3 The diagram illustrates the NHWC output result of the data synthesis unit 104 of the data fusion synchronization output circuit according to a preferred embodiment of this application.

[0015] Figure 4 The diagram shows a circuit block diagram of a data fusion synchronization output circuit according to a preferred embodiment of this application.

[0016] Figure 5 The diagram illustrates a data fusion and synchronous output method according to a preferred embodiment of this application.

[0017] Symbol explanation:

[0018] ADC analog-to-digital converter; CCAP camera image capture interface control circuit; 101 control circuit; 102 buffer synchronization storage circuit; 103 data processing and selection unit; 104 data synthesis unit; R red pixel; G green pixel; B blue pixel; HSYNC horizontal synchronization signal; VSYNC vertical synchronization signal; PCLK pixel clock signal. Detailed Implementation

[0019] Reference will now be made in detail to exemplary embodiments of this application, which are illustrated in the accompanying drawings. Where possible, the same component symbols are used in the drawings and description to refer to the same or similar parts. Furthermore, the exemplary embodiments are merely one way of implementing the design concept of this application, and the following examples are not intended to limit this application.

[0020] Figure 1 The diagram illustrates a circuit block diagram of a data fusion synchronization output circuit according to a preferred embodiment of this application. Please refer to... Figure 1 For ease of explanation, the block diagram of this data fusion synchronization output circuit in this embodiment also includes a Camera Capture Interface Controller (CCAP) circuit and an analog-to-digital converter (ADC). The data output by the CCAP and ADC are two different types of heterogeneous data. Such heterogeneous data generally uses different mechanisms, making synchronous output difficult. In this embodiment, the data fusion synchronization output circuit not only synchronizes the aforementioned heterogeneous data but also fuses it into the data structure required by the convolutional neural network.

[0021] In this embodiment, an image output circuit and an analog-to-digital converter (ADC) are included. The camera image capture interface control circuit (CCAP) outputs a first pixel data R, a second pixel data G, and a third pixel data B. The ADC converts an external analog signal, such as the voltage output from a light, heat, humidity, or smoke sensor, into digital data. It can be seen that in this embodiment, the digital data converted from the voltages (analog signals) of these sensors is essentially heterogeneous data completely unrelated to the pixel data.

[0022] In this embodiment, the data fusion synchronization output circuit includes a control circuit 101, a buffer synchronization storage circuit 102, a data processing selection unit 103, and a data synthesis unit 104. The input port of the control circuit 101 is coupled to the camera image capture interface control circuit (CCAP) to receive, for example, a horizontal synchronization signal HSYNC, a vertical synchronization signal VSYNC, and a pixel clock signal PCLK. Thus, the control circuit 101 can know the timing of the CCAP outputting red pixel R, green pixel G, and blue pixel B. To synchronize the sensor data with the output of red pixel R, green pixel G, and blue pixel B, the trigger control port of the control circuit 101 is coupled to the analog-to-digital converter (ADC). Based on the horizontal synchronization signal HSYNC, the vertical synchronization signal VSYNC, and the pixel clock signal PCLK, the trigger control port of the control circuit 101 outputs a trigger signal (ADC trigger) to trigger the ADC to perform analog-to-digital conversion and output digital data.

[0023] In this embodiment, the buffer synchronization storage circuit 102 has four storage fields, which are used to temporarily store red pixel R, green pixel G, blue pixel B, and digital data A. Since the control circuit 101 triggers the analog-to-digital converter ADC by means of the horizontal synchronization signal HSYNC, the vertical synchronization signal VSYNC, and the pixel clock signal PCLK, the digital data A temporarily stored here is actually synchronized with pixels R, G, and B.

[0024] The data operation selection unit 103 is coupled to the control port of the control circuit 101 and the buffer synchronization storage circuit 102. In this embodiment, the control circuit 101 determines what operation to perform on the data temporarily stored in the buffer synchronization storage circuit 102 based on the nature of the subsequent convolutional neural network. For example, if the subsequent convolutional neural network is to be trained, the red pixel R, green pixel G, blue pixel B, and digital data A need to be normalized beforehand. If the subsequent convolutional neural network is a trained neural network processing unit, and the red pixel R, green pixel G, blue pixel B, and digital data A need to be judged, then in addition to normalization, quantization into INT8 form is also required. Therefore, the control circuit 101 controls the data operation selection unit 103 to perform a specified operation on the data received from the buffer synchronization storage circuit 102 and outputs a specified operation data.

[0025] The data synthesis unit 104, coupled to the control port of the control circuit, and the data operation selection unit 103, sequentially receive the specified operation data. In this embodiment, the control circuit 101 determines the arrangement of the specified operation data of the data synthesis unit 104 based on the nature of the subsequent convolutional neural network. For example, if computation is performed using a graphics processing unit (GPU) or convolution operation, the channel-first (NCHW) format is more efficient; if computation is performed using a tensor processing unit (TPU), the channel-last (NHWC) format achieves better performance. Therefore, the control circuit 101 determines the specified order in which the data synthesis unit 104 arranges the specified operation data and outputs it to the convolutional neural network according to the actual situation.

[0026] Figure 2 The diagram illustrates the NCHW output result of the data synthesis unit 104 of a data fusion synchronous output circuit according to a preferred embodiment of this application. Please refer to... Figure 2 Those skilled in the art will see that the specified format data is in the channel-first (NCHW) manner. Each pixel channel has nine data points, and the analog-to-digital conversion channel A also has nine data points. The pixel data R, pixel data G, pixel data B, and digital data A within the channel are sequentially arranged according to the order of R channel, G channel, B channel, and A channel.

[0027] Figure 3 The diagram illustrates the NHWC output result of the data synthesis unit 104 of a data fusion synchronization output circuit according to a preferred embodiment of this application. Please refer to... Figure 3 Those skilled in the art will see that the specified format data is arranged in the channel-after-the-end (NHWC) manner. Each pixel channel has nine data points, and the analog-to-digital conversion channel A also has nine data points. Here, the pixel data R, pixel data G, pixel data B, and the first data of digital data A in each channel are taken out in sequence, arranged, and then the second data of pixel data R, pixel data G, pixel data B, and digital data A in each channel are taken out in sequence, arranged, and then the third data of pixel data R, pixel data G, pixel data B, and digital data A in each channel are taken out in sequence, and so on, until all the data in the channel is taken out, and the arrangement is completed.

[0028] Figure 4 The diagram illustrates a circuit block diagram of a data fusion synchronization output circuit according to a preferred embodiment of this application. Please refer to... Figure 4In this embodiment, the data processing selection unit 103 is implemented by a first selection circuit SW1, a second selection circuit SW2, a third selection circuit SW3, a normalization operation unit (NORM), and a quantization operation unit (QNT). Assuming the convolutional neural network is used for training, the control circuit 101 controls the first selection circuit SW1 through a control port, electrically connecting the input port of the first selection circuit SW1 to the first output port of the first selection circuit SW1, allowing data from the buffer synchronization storage circuit 102 to be imported into the normalization operation unit (NORM). Additionally, the control circuit 101 controls the second selection circuit SW2 through a control port, electrically connecting the input port of the second selection circuit SW2 to the second output port of the second selection circuit SW2; simultaneously, the control circuit 101 controls the third selection circuit SW3 through a control port, electrically connecting the second input port of the third selection circuit SW3 to the output port of the third selection circuit SW3, allowing data output from the normalization operation unit (NORM) to be transferred to the data synthesis unit 104.

[0029] Similarly, assuming the convolutional neural network is used for judgment, the control circuit 101 controls the first selection circuit SW1 through the control port, electrically connecting the input port of the first selection circuit SW1 to the first output port of the first selection circuit SW1, allowing the data from the buffer synchronization storage circuit 102 to be imported into the normalization operation unit NORM. Additionally, the control circuit 101 controls the second selection circuit SW2 through the control port, electrically connecting the input port of the second selection circuit SW2 to the first output port of the second selection circuit SW1, allowing the data output by the normalization operation unit NORM to be transferred to the quantization operation unit QNT for quantization. Simultaneously, the control circuit 101 controls the third selection circuit SW3 through the control port, electrically connecting the first input port of the third selection circuit SW3 to the output port of the third selection circuit SW1, allowing the data output by the quantization operation unit QNT to be transferred to the data synthesis unit 104.

[0030] Similarly, assuming direct data transmission is required, control circuit 101 controls the first selection circuit SW1 via its control port, electrically connecting the input port of the first selection circuit SW1 to its second output port, allowing data from the buffer synchronization storage circuit 102 to be imported into the third input port of the third selection circuit SW3. Furthermore, control circuit 101 controls the third selection circuit SW3 via its control port, electrically connecting the third input port of the third selection circuit SW3 to the output port of the third selection circuit SW1, allowing data from the buffer synchronization storage circuit 102 to be directly transferred to the data synthesis unit 104.

[0031] Although the above embodiment uses three selection circuits SW1, SW2, and SW3 to implement the data operation selection unit 103, those skilled in the art should know that the data operation selection unit 103 can be implemented in different ways depending on the design. For example, if the first selection circuit SW1 and the second selection circuit SW2 are removed, and only the third selection circuit SW3 is used, and the first input port of the third selection circuit SW3 is coupled to the normalization unit NORM; the output of the normalization unit NORM is coupled to the input of the quantization unit QNT; the second input port of the third selection circuit SW3 is coupled to the quantization unit QNT; and the third input port of the third selection circuit SW3 is coupled to the buffer synchronization storage circuit 102, the above-mentioned versatility can still be achieved. However, the disadvantage is that the normalization unit NORM and the quantization unit QNT need to operate continuously, which does not conform to power saving planning. Since design is a matter of trade-offs, this application is not limited to the above embodiment and will not be elaborated here.

[0032] The above embodiments can be summarized into a data fusion and synchronous output method. Figure 5 The diagram illustrates a flowchart of a data fusion and synchronous output method according to a preferred embodiment of this application. Please refer to... Figure 5 In this embodiment, the data fusion and synchronization output method includes the following steps:

[0033] Step S501: Begin.

[0034] Step S502: According to a received clock, trigger an analog-to-digital converter (ADC) to output digital data. As in the above embodiment, the control circuit uses the horizontal synchronization signal HSYNC, the vertical synchronization signal VSYNC, and a pixel clock signal PCLK to periodically trigger the ADC to output digital data.

[0035] Step S503: According to the receiving clock, temporarily store a first pixel data, a second pixel data, a third pixel data, and the digital data to a buffer synchronization storage circuit. As in the above embodiment, the first pixel data R, the second pixel data G, the third pixel data B, and the digital data A can be synchronized by a triggering mechanism.

[0036] Step S504: According to the data requirements of a convolutional neural network, perform a specified operation on the first pixel data, the second pixel data, the third pixel data, and the digital data. As in the above embodiment, during network training or inference, a normalization operation can be performed to normalize the original data value range to floating-point values ​​between [-1,1] or [0,1]. Adding this operation can reduce the CPU execution time. For microcontrollers with neural processing unit (NPU) inference capabilities, the data type is generally required to be INT8. Therefore, the normalized floating-point data needs to be quantized again to INT8 values.

[0037] Step S505: According to the data requirements of the convolutional neural network, perform data fusion on the specified processed first pixel data, second pixel data, third pixel data, and digital data, and output data in a specified format. Please refer to [reference needed] for the fusion method. Figure 2 , Figure 3 The above embodiments and descriptions will not be repeated here.

[0038] In summary, a preferred embodiment of this application provides a data fusion and synchronous output method that effectively integrates data from different sources and performs computation and data fusion according to the requirements of a convolutional neural network (CNN). By triggering the analog-to-digital converter to output digital data according to the receiving clock and temporarily storing multiple pixel data and digital data in a buffer synchronous storage circuit, the real-time performance and synchronization of the data are ensured. Furthermore, data can be flexibly processed and output according to different data arrangement formats (such as NCHW or NHWC), improving the efficiency and flexibility of the system. During training or inference, normalization and quantization operations are performed on the first, second, and third pixel data and digital data, which helps to improve model performance.

[0039] It should be understood that the examples and embodiments described herein are for illustrative purposes only, and various modifications or changes thereto will be suggested to those skilled in the art and will be included within the spirit and scope of this application and the scope of the appended claims.

Claims

1. A data fusion synchronous output circuit, suitable for organizing heterogeneous data and synchronously outputting it to a convolutional neural network, characterized in that, A heterogeneous data output circuit includes: A camera image capture interface control circuit is used to output a first pixel data, a second pixel data, and a third pixel data; and An analog-to-digital converter is used to convert an external analog signal into digital data; The data fusion synchronization output circuit includes: A control circuit includes an input port, a control port, and a trigger control port. The input port of the control circuit is coupled to the camera image capture interface control circuit to receive at least one synchronization signal and a pixel clock signal. The trigger control port of the control circuit is coupled to the analog-to-digital converter. Based on the synchronization signal and the pixel clock signal, the trigger control port of the control circuit outputs a trigger signal to trigger the analog-to-digital converter to perform analog-to-digital conversion to output the digital data. A buffer synchronous storage circuit, coupled to the camera image capture interface control circuit and the analog-to-digital converter, is used to receive the first pixel data, the second pixel data, the third pixel data and the digital data according to a receiving clock. A data operation selection unit is coupled to the control port of the control circuit and the buffer synchronization storage circuit. The data operation selection unit determines, according to the control of the control circuit, to perform a specified operation on the data received from the buffer synchronization storage circuit and outputs a specified operation data. A data synthesis unit, coupled to the control port of the control circuit and the data operation selection unit, sequentially receives the specified operation data, and according to the control of the control circuit, arranges the received multiple specified operation data in a specified order and outputs them to the convolutional neural network.

2. The data fusion synchronous output circuit as described in claim 1, characterized in that, The specified order includes accessing height and width data within a single channel according to channel priority.

3. The data fusion synchronous output circuit as described in claim 1, characterized in that, The specified order includes sequentially accessing the height and width data of each channel in a channel-after-channel manner until the data storage of each channel is complete.

4. The data fusion synchronous output circuit as described in claim 1, characterized in that, The data processing selection unit includes: A first selection circuit includes a control port, an input port, a first output port, and a second output port, wherein the control port of the first selection circuit is coupled to the control port of the control circuit, the input port of the first selection circuit is coupled to the buffer synchronization storage circuit, and the second output port of the first selection circuit is coupled to the data synthesis unit; and A normalization operation unit includes an input port and an output port, wherein the input port of the normalization operation unit is coupled to the first output port of the first selection circuit, and the output port of the normalization operation unit is coupled to the data synthesis unit.

5. The data fusion synchronous output circuit as described in claim 4, characterized in that, The data processing selection unit further includes: A second selection circuit includes a control port, an input port, a first output port, and a second output port, wherein the control port of the second selection circuit is coupled to the control port of the control circuit, and the input port of the second selection circuit is coupled to the output port of the normalization operation unit. A quantization unit includes an input port and an output port, wherein the input port of the quantization unit is coupled to the first output port of the second selection circuit; and A third selection circuit includes a control port, a first input port, a second input port, a third input port, and an output port. The control port of the third selection circuit is coupled to the control port of the control circuit. The first input port of the third selection circuit is coupled to the output port of the quantization unit. The second input port of the third selection circuit is coupled to the second output port of the second selection circuit. The third input port of the third selection circuit is coupled to the second output port of the first selection circuit. The output port of the third selection circuit is coupled to the data synthesis unit.

6. A data fusion and synchronous output method, characterized in that, According to a receiving clock, an analog-to-digital converter is triggered to output digital data; According to the receiving clock, a first pixel data, a second pixel data, a third pixel data, and the digital data are temporarily stored in a buffer synchronization storage circuit; According to the data requirements of a convolutional neural network, a specified operation is performed on the first pixel data, the second pixel data, the third pixel data, and the digital data to obtain the first pixel data, the second pixel data, the third pixel data, and the digital data after the specified operation. as well as According to the data requirements of the convolutional neural network, the first pixel data, the second pixel data, the third pixel data, and the digital data after specified operations are fused to output data in a specified format.

7. The data fusion and synchronous output method as described in claim 6, characterized in that, The specified format data is arranged in a channel-priority manner, sequentially arranging all the first pixel data, the second pixel data, the third pixel data, and the digital data within the channel.

8. The data fusion and synchronous output method as described in claim 6, characterized in that, The specified format data is obtained by sequentially extracting and arranging the first pixel data, the second pixel data, the third pixel data, and the first to last data of the digital data arrangement in each channel according to the channel-last method, until all data in the channel is extracted.

9. The data fusion and synchronous output method as described in claim 6, characterized in that, According to the data requirements of the convolutional neural network, the specified operation is performed on the first pixel data, the second pixel data, the third pixel data, and the digital data to obtain the first pixel data, the second pixel data, the third pixel data, and the digital data after the specified operation, including: When the convolutional neural network needs to be trained: A normalization operation is performed on the first pixel data, the second pixel data, the third pixel data, and the digital data.

10. The data fusion and synchronous output method as described in claim 6, Its features are, Specifically, according to the data requirements of the convolutional neural network, the specified operation is performed on the first pixel data, the second pixel data, the third pixel data, and the digital data to obtain the first pixel data, the second pixel data, the third pixel data, and the digital data after the specified operation, including: When the convolutional neural network needs to make a decision... Perform a normalization operation on the first pixel data, the second pixel data, the third pixel data, and the digital data; and A quantization operation is performed on the normalized first pixel data, the second pixel data, the third pixel data, and the digital data.

Images