Video decoder, video processing device and electronic equipment

Abstract

The embodiment of the invention provides a video decoder. The video decoder comprises a video decoding circuit and a data preprocessing circuit, the output end of the video decoding circuit is connected with the input end of the data preprocessing circuit; the video decoding circuit is used for decoding a to-be-processed video to obtain a to-be-processed image; the data preprocessing circuit is used for preprocessing the to-be-processed image to obtain target data; the target data is used for being processed by a neural network processor. The embodiment of the invention further provides a video processing device and electronic equipment.

Description

[0001] This application is a divisional application of the invention patent application filed on August 4, 2021, with application number "202110893162.2" and application title "Video Decoder, Video Processing Device and Electronic Equipment" at the State Intellectual Property Office of China. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to a video decoder, video processing device, and electronic device. Background Technology

[0003] Intelligent video analytics refers to the use of computer vision algorithms or artificial intelligence reasoning to process each frame of a video stream to identify objects or behaviors of interest to the user. This includes, but is not limited to, object recognition, face detection, intrusion detection, liveness detection, orientation detection, object classification, and behavior counting.

[0004] Currently, the intelligent video analytics process involves numerous processing steps, requiring frequent readings of data from storage units, which increases system bandwidth consumption and reduces system performance. Summary of the Invention

[0005] This application provides a video decoder, a video processing device, and an electronic device.

[0006] The technical solution of this application is implemented as follows: In a first aspect, a video decoder is provided, comprising: a video decoding circuit and a data preprocessing circuit; The output terminal of the video decoding circuit is connected to the input terminal of the data preprocessing circuit; The video decoding circuit is used to decode the video to be processed to obtain the image to be processed. The data preprocessing circuit is used to preprocess the image to be processed to obtain target data; the target data is used for processing by the neural network processor.

[0007] In a second aspect, a video processing apparatus is provided, including a video decoder and a neural network processor; The video decoder includes a video decoding circuit and a data preprocessing circuit; The output terminal of the video decoding circuit is connected to the input terminal of the data preprocessing circuit; The video decoding circuit is used to decode the video to be processed to obtain the image to be processed. The data preprocessing circuit is used to preprocess the image to be processed to obtain target data, and store the target data in the storage unit; The neural network processor is used to obtain the target data from the storage unit and process the target data.

[0008] Thirdly, an electronic device is provided, including a storage unit; and the video processing apparatus provided above. This application provides a video decoder, comprising a video decoding circuit and a data preprocessing circuit. The output of the video decoding circuit is connected to the input of the data preprocessing circuit. The video decoding circuit decodes the video to be processed to obtain an image to be processed. The data preprocessing circuit preprocesses the image to be processed to obtain target data. The target data is used for processing by a neural network processor. It can be seen that the embodiment provided in this application integrates the data preprocessing circuit into the video decoder, directly preprocessing the decoded image to be processed and converting it into data that the neural network processor can directly use. This reduces the number of read operations performed on the storage unit, lowers system bandwidth usage, and improves system processing performance. Attached Figure Description

[0009] Figure 1 A schematic diagram of an intelligent video analysis process in a related art provided in this application embodiment. Figure 1 ; Figure 2 A schematic diagram of an intelligent video analysis process in a related art provided in this application embodiment. Figure 2 ; Figure 3 A schematic diagram of the structural composition of a video decoder provided in this application embodiment. Figure 1 ; Figure 4 A schematic diagram of the structural composition of a video decoder provided in this application embodiment. Figure 2 ; Figure 5 A schematic diagram of the structural composition of a video decoder provided in this application embodiment. Figure 3 ; Figure 6 A schematic diagram of the structure of a data preprocessing circuit provided in an embodiment of this application. Figure 1 ; Figure 7 A schematic diagram of the structure of a data preprocessing circuit provided in an embodiment of this application. Figure 2 ; Figure 8 A schematic diagram of the structure of a data preprocessing circuit provided in an embodiment of this application. Figure 3 ; Figure 9 A schematic diagram of the structural composition of a video decoder provided in this application embodiment. Figure 4 ; Figure 10 This is a schematic diagram of the structural composition of a video processing device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structural composition of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this invention.

[0011] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0012] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0013] In practical applications, intelligent video analysis typically requires decoding the video using a video decoder to obtain continuous image frames. These continuous image frames are then processed by a neural network processor using computer vision algorithms or artificial intelligence reasoning to obtain the analysis results.

[0014] Typically, video decoders and neural network processors are designed by engineers from different fields. The data format obtained by the video decoder is not the same as the data format that the neural network processor can process. For example, the output data format of the video decoder is in luminance-chrominance (YUV) image format, while the input data of the neural network processor is usually in red-green-blue (RGB) image format. Since the data between the video decoder and the neural network processor cannot be directly exchanged, data preprocessing is required for the output data of the video decoder.

[0015] In practical applications, videos are usually compressed and encoded before being stored in storage units. The compressed video data can be called a video stream.

[0016] refer to Figure 1 The diagram illustrates an intelligent video analysis process in a related technology. Figure 1 The first step in intelligent video analysis by a video processing device is to read the video stream from the storage unit (see reference). Figure 1 Channel 1 (as shown) decodes the video stream using a video decoder (also known as decompression) to obtain consecutive image frames. These consecutive image frames are typically in YUV image format, and the video processing device writes them to a storage unit (see reference). Figure 1 (Channel 2 shown). Next, the video processing device performs data preprocessing on the decoded consecutive image frames, converting them into data that can be directly used by the neural network processor. Data preprocessing includes, but is not limited to, frame rate downsampling, image format conversion, image size adjustment, and data type conversion.

[0017] During data preprocessing, the video processing device first needs to read the decoded consecutive image frames from the storage unit (see reference). Figure 1 Channel 3 (as shown), and after preprocessing, the processing result is written to the storage unit (see reference). Figure 1 Channel 4 is shown. Finally, the neural network processor reads data from the storage unit (see reference). Figure 1 Channel 5 (as shown) performs artificial intelligence algorithm calculations or artificial intelligence inference, and after obtaining the calculation results, writes the calculation results into the storage unit (see reference). Figure 1 Channel 6 is shown.

[0018] In related technical solutions, the data preprocessing process is quite complex and needs to be completed in multiple steps. (Reference) Figure 2 As shown, data preprocessing may need to be performed in two stages. Each stage requires reading data from the storage unit, and the processing result is written back to the storage unit after processing. In this case, the video processing device will frequently read data from the storage unit, consuming a significant amount of system bandwidth. In particular, with a fixed system bandwidth, the more read and write operations the data preprocessing performs on the storage unit, the lower the overall system processing capacity.

[0019] In practical applications, data preprocessing can be implemented in hardware or software. Software-based data preprocessing is significantly slower than hardware-based preprocessing. However, even with hardware-based preprocessing, frequent reads and writes to storage units still consume substantial system bandwidth, impacting system processing performance.

[0020] Based on this, embodiments of this application provide a video decoder, which includes a video decoding circuit and a data preprocessing circuit; wherein the output terminal of the video decoding circuit is connected to the input terminal of the data preprocessing circuit; the video decoding circuit is used to decode the video to be processed to obtain an image to be processed; the data preprocessing circuit is used to preprocess the image to be processed to obtain target data; the target data is used for processing by a neural network processor. It can be seen that the embodiments provided in this application can integrate the data preprocessing circuit into the video decoder, directly preprocessing the decoded image to be processed through the data preprocessing circuit, converting it into data that can be directly used by the neural network processor. This reduces the number of read operations performed on the storage unit, reduces system bandwidth usage, and improves system processing performance.

[0021] To facilitate understanding of the technical solutions of the embodiments of this application, the technical solutions of this application are described in detail below through specific embodiments. The above-mentioned related technologies are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. The embodiments of this application include at least some of the following contents.

[0022] One embodiment of this application provides a video decoder, referencing... Figure 3 As shown, the video decoder 30 may include: a video decoding circuit 31 and a data preprocessing circuit 32; wherein, The output of the video decoding circuit 31 is connected to the input of the data preprocessing circuit 32; The video decoding circuit 31 is used to decode the video to be processed to obtain the image to be processed. The data preprocessing circuit 32 is used to preprocess the image to be processed to obtain target data; the target data is used for processing by the neural network processor.

[0023] It is understood that the embodiments of this application employ hardware to implement the video decoding and data preprocessing processes. Specifically, the video is decoded using a hardware-based video decoding circuit 31 to obtain the image to be processed. Here, the image to be processed may include multiple consecutive frames.

[0024] Furthermore, the data preprocessing process for the image to be processed can be implemented through the hardware-based data preprocessing circuit 32. Here, data preprocessing refers to performing neural network data preprocessing on the image to be processed, converting it into data that the neural network processor can directly process. For example, converting integer data in the image to include floating-point or half-floating-point data.

[0025] In this embodiment of the application, the output terminal of the video decoding circuit 31 is connected to the input terminal of the data preprocessing circuit 32. In this way, the image to be processed after being decoded by the video decoding circuit 31 can be directly input to the data preprocessing circuit 32 for processing, without having to write the decoded image to be processed into the storage unit and then read it from the storage unit for preprocessing.

[0026] In other words, in the embodiments provided in this application, the data preprocessing circuit 32 can be integrated into the video decoder 30. The data preprocessing circuit 32 can directly preprocess the decoded image to be processed, converting it into data that can be directly used by the neural network processor. This reduces the number of read operations from the storage unit, reduces the use of system bandwidth, and improves processing performance.

[0027] In some embodiments, reference Figure 4 The video decoder 30 shown may include a data preprocessing circuit 32, which may include an image format conversion sub-circuit 321. The input terminal of the image format conversion sub-circuit 321 is connected to the output terminal of the video decoding circuit 31; Image format conversion sub-circuit 321 is used to convert the image format of the image to be processed from a first image format to a second image format to obtain first data to be processed; the first data to be processed is target data.

[0028] In this embodiment, the image format conversion sub-circuit 321 is directly connected to the video decoding circuit 31, and the image format conversion sub-circuit 321 can receive the image to be processed output by the video decoding circuit 31. Thus, the image format of the image to be processed is converted through the hardware-based image format conversion sub-circuit 321.

[0029] It should be noted that the first image format and the second image format are different. Specifically, the first image format can be a YUV image format and the second image format can be an RGB image format; or, the first image format can be a RAW image format and the second image format can be an RGB image format. This application does not limit the first image format or the second image format.

[0030] For example, the conversion formula for converting a YUV image format to an RGB image format is shown in Formula 1-1.

[0031]

[0032] Where Y represents the luminance component of the image, and U and V represent the chrominance components. R represents the red luminance component, G represents the green luminance component, and B represents the blue luminance component.

[0033] In some embodiments, the image format conversion sub-circuit 321 may include multiple multipliers and multiple adders for converting between two different image formats as shown in Formula 1-1. The image format conversion sub-circuit 321 may also include a conversion parameter register for storing conversion parameters required during the image format conversion process (e.g., parameters a1~a3, b1~b3, and c1~c3 in Formula 1-1).

[0034] Based on this, the image format conversion sub-circuit 321 can realize the conversion between different image formats through hardware circuit structure under the configuration of the conversion parameter register. Therefore, the video decoder 30 provided in this embodiment can directly perform image format conversion processing on the decoded image to be processed, converting it into an image format that can be directly used by the neural network processor. This reduces the number of read operations from the storage unit, reduces system bandwidth usage, and improves processing performance.

[0035] In some embodiments, the image format conversion sub-circuit 321 can perform format conversion processing on each frame of the image to be processed.

[0036] It should be noted that the current video transmission rate is 30-60 frames per second (FPS). Since the image content of adjacent image frames is basically the same, with only minor changes, it is not necessary to perform data preprocessing on every frame of the image to be processed. In this embodiment, the frame rate of the decoded image to be processed can be downsampled, and a lower frame rate can be obtained by skipping frames. For example, skipping frames can be used to obtain images of 1, 2, 5, or 10 frames per second for data preprocessing and subsequent neural network processing.

[0037] In some embodiments, the frame rate of the image to be processed can be downsampled in software.

[0038] In some embodiments, frame rate downsampling of the image to be processed can be performed using hardware circuitry. Specifically, refer to... Figure 5 The video decoder shown may include a data preprocessing circuit 32, which may include an image format conversion sub-circuit 321 and a frame rate sampling sub-circuit 322. The input terminal of the frame rate sampling sub-circuit 322 is connected to the output terminal of the video decoding circuit 31; The output of the frame rate sampling sub-circuit 322 is connected to the input of the image format conversion sub-circuit 321; The frame rate sampling sub-circuit 322 is used to perform frame rate downsampling on the image to be processed to obtain a sampled image. Image format conversion sub-circuit 321 is used to convert the image format of the sampled image from a first image format to a second image format to obtain first data to be processed; the first data to be processed is target data.

[0039] It is understood that the data preprocessing circuit 32 may include a frame rate sampling sub-circuit 322. The frame rate sampling sub-circuit 322 performs frame rate downsampling on the decoded continuous image frames (i.e., the image to be processed) to obtain a sampled image. Then, the sampled image is input to the image format conversion sub-circuit 321 for image format conversion processing to obtain target data that can be directly processed by the neural network processor.

[0040] In some embodiments, the frame rate sampling sub-circuit 322 may include a frame rate configuration register. The frame rate configuration register is used to store frame rate configuration information, and the actual frame rate used can be set through this frame rate configuration register.

[0041] As can be seen, the video decoder provided in this application embodiment can preprocess data through hardware circuits. Compared with software processing, the scheduling method is simple, the calculation speed is fast, and the efficiency of data preprocessing can be greatly improved.

[0042] Based on the foregoing embodiments, in some embodiments, reference is made to... Figure 6 The data preprocessing circuit 32 shown may further include: a data type conversion sub-circuit 323; wherein, The input terminal of the data type converter 323 circuit is connected to the output terminal of the image format converter 321 circuit; The data type conversion sub-circuit 323 is used to convert the data type of the first data to be processed from a first data type to a second data type to obtain the second data to be processed; the second data to be processed is the target data.

[0043] In some embodiments, the first data type includes an integer; the second data type includes a floating-point or half-floating-point type.

[0044] In practical applications, neural network processors typically process floating-point or half-floating-point data, while the data type in the image to be processed after video decoding is typically integer data. Therefore, the data preprocessing circuit 32 provided in this embodiment may further include a data type conversion sub-circuit 323 to convert data types.

[0045] In some embodiments, the data type conversion sub-circuit 323 may include one or more of a floating-point conversion sub-circuit, a half-floating-point conversion sub-circuit, and a novel floating-point conversion sub-circuit. In practical applications, one of these conversion sub-circuits can be determined to be used based on the data requirements of the neural network algorithm running on the neural network processor.

[0046] For example, when the neural network processor can process floating-point data types, the floating-point conversion sub-circuit can be turned on, while the half-floating-point conversion sub-circuit and the novel floating-point conversion sub-circuit can be turned off. In this way, the first data to be processed after image format conversion can be input into the floating-point conversion sub-circuit for data type conversion processing, resulting in the target data that the neural network processor can directly process.

[0047] As can be seen, the data preprocessing circuit provided in this application embodiment can support the conversion of different data types. Furthermore, this application embodiment achieves data type conversion through a hardware circuit structure with a data type conversion sub-circuit. Compared to software processing methods, this approach is simpler to schedule, faster in computation, and can significantly improve the efficiency of data preprocessing.

[0048] In some embodiments, reference Figure 7 The data preprocessing circuit 32 shown may further include: a numerical adjustment sub-circuit 324; wherein, The input terminal of the numerical adjustment sub-circuit 324 and the output terminal of the data type conversion sub-circuit 323; The numerical adjustment sub-circuit 324 is used to map the second data to be processed to a preset numerical range to obtain the third data to be processed; the third data to be processed is the target data.

[0049] Here, the numerical adjustment sub-circuit 324 is directly connected to the data type conversion sub-circuit 323, which can adjust the value of the second data to be processed output by the data type conversion sub-circuit 323 so that the second data to be processed is within a certain numerical range and / or numerical range interval, so as to facilitate the subsequent processing by the neural network processor.

[0050] In some embodiments, the numerical adjustment sub-circuit 324 may include a normalization processing circuit and / or a shaping processing circuit. The normalization processing circuit normalizes the second data to be processed, ensuring that the value of the second data falls within a certain range, such as [-1, 1], [-0.5, 0.5], etc. The shaping processing circuit shapes the second data to be processed, changing the range of its values. For example, shaping may involve dividing the value of the second data to be processed by a certain parameter, thereby changing the range of its values.

[0051] In some embodiments, when the numerical adjustment sub-circuit 324 includes both a normalization processing circuit and a shaping processing circuit, the specific connection structure may be: The input terminal of the normalization processing circuit is connected to the output terminal of the data type conversion sub-circuit 323; The output of the normalization circuit is connected to the input of the shaping circuit.

[0052] In other words, the second data to be processed after being processed by the data type conversion sub-circuit 323 can first be normalized by the normalization processing circuit, and then shaped by the shaping processing circuit.

[0053] In other embodiments, the input of the shaping circuit is connected to the output of the data type conversion sub-circuit 323; the output of the shaping circuit is connected to the input of the normalization circuit.

[0054] In other words, the second data to be processed after being processed by the data type conversion sub-circuit 323 can also be first shaped by the shaping circuit and then normalized by the normalization circuit.

[0055] Therefore, the data preprocessing circuit provided in this application embodiment can perform normalization and / or shaping processing on the second data to be processed through hardware circuit structure. Compared with software processing, the scheduling method is simple, the calculation speed is fast, and the efficiency of data preprocessing can be greatly improved.

[0056] Based on the above embodiments, in some embodiments, reference is made to... Figure 8 The data preprocessing circuit 32 shown may further include a size adjustment sub-circuit 325; The input terminal of the size adjustment sub-circuit 325 is connected to the output terminal of the image format conversion sub-circuit 321; The output of the size adjustment sub-circuit 325 is connected to the input of the data type conversion sub-circuit 323; The size adjustment sub-circuit 325 is used to adjust the size of the image to be processed in the second image format to obtain the first data to be processed.

[0057] In this embodiment of the application, to facilitate processing by the neural network processor, the size of the image to be processed after image format conversion can be adjusted. Specifically, the size adjustment of the image to be processed after image format conversion can be achieved through hardware circuit structure.

[0058] In some embodiments, the size adjustment sub-circuit 325 may include a size scaling circuit and / or a trimming circuit; wherein... A size scaling circuit is used to enlarge or reduce the size of the image to be processed in the second image format to obtain the first data to be processed. The cropping circuit is used to acquire images within a preset range from the image to be processed in the second image format, and obtain the first data to be processed.

[0059] In some embodiments, the size scaling circuit can scale the length and width of the image to be processed in the second image format, and the scaling size can be determined according to the neural network algorithm run by the neural network processor.

[0060] In some embodiments, typical neural network algorithms require processing based on square images. When the length and width of the image to be processed in a second image format are inconsistent, a cropping circuit is also needed to crop the image in the second image format. For example, the cropping circuit can retain the image center and remove the content at the left and right ends along the longer direction, making the cropped image a square. If the neural network algorithm supports rectangular images, this circuit can be set to a non-operating state.

[0061] In some embodiments, in order to satisfy different neural network algorithms, the number of size scaling circuits may include multiple circuits, which are respectively set at the input and output of the clipping circuit.

[0062] For example, the size scaling circuit may include a first size scaling circuit and a second size scaling circuit. The connection relationship is as follows: The input terminal of the first size scaling circuit is connected to the output terminal of the image format conversion sub-circuit 321; The output of the first size scaling circuit is connected to the input of the trimming circuit; The output of the trimming circuit is connected to the input of the second size scaling circuit.

[0063] When the neural network algorithm needs to scale an image before cropping it, it can send enable signals to the first scaling circuit and the cropping circuit, making them active while the second scaling circuit is inactive. Similarly, when the neural network algorithm needs to crop an image before scaling it, it can send enable signals to the second scaling circuit and the cropping circuit, making them active while the first scaling circuit is inactive.

[0064] Therefore, the data preprocessing circuit provided in this application embodiment can include processing sub-circuits with different functions, and the working state of the processing sub-circuits can be set according to the needs of the neural network algorithm. In this way, the decoded image to be processed can be converted into a format that the neural network processor can directly use. This reduces the number of read operations from the storage unit, reduces system bandwidth usage, and improves processing performance.

[0065] In some embodiments, the data preprocessing circuit may further include a data format conversion sub-circuit for converting the data format of the first data to be processed, the second data to be processed, or the third data to be processed, from height-width-depth (HWC) format to depth-height-width (CHW) format, or from CHW format to HWC format.

[0066] In some embodiments, reference Figure 9 The video decoder 30 shown in this application embodiment includes multiple data preprocessing circuits 32. Multiple data preprocessing circuits 32 are connected in parallel, and the multiple data preprocessing circuits 32 process the image to be processed in parallel.

[0067] In other words, the input terminals of multiple data preprocessing circuits 32 are connected to the video decoding circuit 31 to receive the image to be processed output by the video decoding circuit 31.

[0068] In some embodiments, the image to be processed can be divided into multiple parts and output to multiple data preprocessing circuits 32 for parallel processing, which can increase the data preprocessing rate and improve the data preprocessing efficiency.

[0069] In some embodiments, the processing sub-circuits included within different data preprocessing circuits 32 may differ, each corresponding to a different neural network algorithm. For example, the neural network processor can run two different neural network algorithms, where the first neural network algorithm can process floating-point data, and the second neural network algorithm can process half-floating-point data. Thus, the decoded data to be processed can be processed by the first data preprocessing circuit including a floating-point switching sub-circuit and the second data preprocessing circuit including a half-floating-point switching sub-circuit, respectively, to obtain data formats that can be directly used by the first and second neural network algorithms.

[0070] In summary, the embodiments provided in this application can push the data preprocessing process forward to the video decoder, reducing the number of read operations on the storage unit, reducing system bandwidth usage, and improving processing performance. Furthermore, the embodiments provided in this application employ a pure hardware circuit structure to implement the data preprocessing process, which is faster and simpler to schedule compared to pure software processing.

[0071] One embodiment of this application also provides a video processing apparatus, see reference. Figure 10 As shown, the video processing device 100 includes a video decoder 30 and a neural network processor 40; The video decoder 30 includes a video decoding circuit 31 and a data preprocessing circuit 32. The output of the video decoding circuit 31 is connected to the input of the data preprocessing circuit 32; The video decoding circuit 31 is used to decode the video to be processed to obtain the image to be processed. The data preprocessing circuit 32 is used to preprocess the image to be processed, obtain the target data, and store the target data in the storage unit; The neural network processor 40 is used to retrieve target data from the storage unit and process the target data.

[0072] In some embodiments, the video processing device may be located in an electronic device, such as a smartphone, tablet, personal calculator, server, or industrial calculator.

[0073] In some embodiments, the storage unit may be a storage device independent of the video processing device. For example, the storage unit may be a static random-access memory (SRAM), a dynamic random-access memory (DRAM), etc. The embodiments of this application do not limit the type of storage unit.

[0074] In this embodiment, the data preprocessing circuit 32 is integrated into the video decoder 30, meaning the data preprocessing function is moved forward to the video decoder. Thus, the image to be processed after being decoded by the video decoding circuit 31 can be directly preprocessed, converted into a format that the neural network processor 40 can directly use, and then written into the storage unit. By reducing one image reading operation from the storage device, the video processing apparatus provided in this embodiment reduces system bandwidth usage and improves system processing capabilities.

[0075] In some embodiments, the image to be processed can also be stored in the aforementioned storage unit. That is, after the video decoding circuit 31 obtains the image to be processed, in addition to transmitting the image to be processed to the data preprocessing circuit 32 for preprocessing, it can also simultaneously write the obtained image to be processed into the storage unit for use in subsequent data processing.

[0076] For example, Figure 9 The video processing device shown includes read / write channels for five storage units. Specifically, the data preprocessing process may include: a video decoder 30 reads a video stream from the storage unit through channel 1; the video decoding circuit 31 in the video decoder 30 decodes the read video stream to obtain a YUV format image to be processed. The video decoder can split the obtained image into two paths: one path is directly written to the storage unit through channel 3, and the other path is directly sent to the built-in data preprocessing circuit 32 of the video decoder 30. After the data preprocessing circuit processes the data, it writes the obtained target data to the storage unit through channel 3. Finally, the neural network processor 40 reads the target data from the storage unit through channel 4 and processes the target data. After the neural network processor completes its processing, it writes the processing result to the storage device through channel 5.

[0077] Therefore, the video processing apparatus provided in this application embodiment can directly preprocess video decoding and neural network data at the video decoder end, converting the decoded image into data that the neural network processor can directly use. This reduces the number of read operations from the storage unit, reduces system bandwidth usage, and improves processing performance.

[0078] This application also provides an electronic device, which can be a smartphone, tablet computer, personal calculator, server, or industrial calculator, etc.

[0079] In this embodiment, the electronic device 1100 may include a video processing device 100 and a storage unit 200. (See reference...) Figure 11 As shown, the video processing device 100 may include a video decoder 30 and a neural network processor 40; The video decoder 30 includes a video decoding circuit 31 and a data preprocessing circuit 32. The output of the video decoding circuit 31 is connected to the input of the data preprocessing circuit 32; The video decoding circuit 31 is used to decode the video to be processed to obtain the image to be processed. The data preprocessing circuit 32 is used to preprocess the image to be processed, obtain the target data, and store the target data in the storage unit 200; The neural network processor 40 is used to acquire target data from the storage unit 200 and process the target data.

[0080] In some embodiments, the storage unit 200 is also used to store the image to be processed.

[0081] It should be understood that the video processing device 100 in this embodiment is the same as the video processing device in the above embodiment, and will not be described again here for the sake of brevity.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0083] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or at least two units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0085] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0086] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0087] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0088] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video decoder, characterized in that, include: Video decoding circuit and data preprocessing circuit; The output terminal of the video decoding circuit is connected to the input terminal of the data preprocessing circuit; The video decoding circuit is used to decode the video to be processed to obtain the image to be processed, and to transmit the image to be processed to the data preprocessing circuit. The data preprocessing circuit is used to receive the image to be processed output by the video decoding circuit, preprocess the received image to obtain target data; the target data is used for processing by the neural network processor.

2. The video decoder according to claim 1, characterized in that, The data preprocessing circuit includes: an image format conversion sub-circuit; The input terminal of the image format conversion sub-circuit is connected to the output terminal of the video decoding circuit; The image format conversion sub-circuit is used to convert the image format of the image to be processed from a first image format to a second image format to obtain first data to be processed; the first data to be processed is the target data or is used to obtain the target data.

3. The video decoder according to claim 1, characterized in that, The data preprocessing circuit includes: an image format conversion sub-circuit and a frame rate sampling sub-circuit; The input terminal of the frame rate sampling sub-circuit is connected to the output terminal of the video decoding circuit; The output of the frame rate sampling sub-circuit is connected to the input of the image format conversion sub-circuit; The frame rate sampling sub-circuit is used to perform frame rate downsampling processing on the image to be processed to obtain a sampled image; The image format conversion sub-circuit is used to convert the image format of the sampled image from a first image format to a second image format to obtain first data to be processed; the first data to be processed is the target data or is used to obtain the target data.

4. The video decoder according to claim 2 or 3, characterized in that, The data preprocessing circuit further includes: a data type conversion sub-circuit; The input terminal of the data type conversion sub-circuit is connected to the output terminal of the image format conversion sub-circuit; The data type conversion sub-circuit is used to convert the data type of the first data to be processed from a first data type to a second data type to obtain the second data to be processed; the second data to be processed is the target data or is used to obtain the target data.

5. The video decoder according to claim 4, characterized in that, The first data type includes integer; the second data type includes floating-point or half-floating-point.

6. The video decoder according to claim 4, characterized in that, The data preprocessing circuit also includes a numerical adjustment sub-circuit; The input terminal of the numerical adjustment sub-circuit and the output terminal of the data type conversion sub-circuit; The numerical adjustment sub-circuit is used to map the second data to be processed to a preset numerical range to obtain a third data to be processed; the third data to be processed is the target data; or... The data preprocessing circuit also includes a size adjustment sub-circuit; The input terminal of the size adjustment sub-circuit is connected to the output terminal of the image format conversion sub-circuit; The output terminal of the size adjustment sub-circuit is connected to the input terminal of the data type conversion sub-circuit; The size adjustment sub-circuit is used to adjust the size of the image to be processed in the second image format to obtain the first data to be processed.

7. The video decoder according to any one of claims 1-3, characterized in that, The number of the data preprocessing circuits includes multiple circuits; Multiple data preprocessing circuits are connected in parallel, and the multiple data preprocessing circuits process the image to be processed in parallel.

8. The video decoder according to claim 7, characterized in that, The image to be processed is divided into multiple parts, and each part of the image to be processed is output to multiple data preprocessing circuits for parallel processing.

9. The video decoder according to any one of claims 1-3, characterized in that, The video decoding circuit is used to read the video to be processed from the storage unit.

10. The video decoder according to any one of claims 1-3, characterized in that, The video decoding circuit is used to transmit the image to be processed to the data preprocessing circuit, and at the same time write the image to be processed into the storage unit.

11. The video decoder according to any one of claims 1-3, characterized in that, The data preprocessing circuit is used to store the target data in a storage unit so that the neural network processor can obtain the target data from the storage unit and process the target data.

12. The video decoder according to any one of claims 1-3, characterized in that, The video decoding circuit and the data preprocessing circuit are respectively implemented in hardware to decode the video to be processed and to preprocess the received image to be processed.

13. A video processing apparatus, characterized in that, Includes video decoders and neural network processors; The video decoder includes a video decoding circuit and a data preprocessing circuit; The output terminal of the video decoding circuit is connected to the input terminal of the data preprocessing circuit; The video decoding circuit is used to decode the video to be processed to obtain the image to be processed, and to transmit the image to be processed to the data preprocessing circuit. The data preprocessing circuit is used to receive the image to be processed output by the video decoding circuit, preprocess the received image to be processed to obtain target data, and store the target data in the storage unit. The neural network processor is used to obtain the target data from the storage unit and process the target data.

14. The video processing apparatus according to claim 13, characterized in that, The video decoding circuit is used to transmit the image to be processed to the data preprocessing circuit, and simultaneously write the image to be processed into the storage unit.

15. The video processing apparatus according to claim 13, characterized in that, The video decoding circuit is used to read the video to be processed from the storage unit.

16. An electronic device, characterized in that, include: Storage unit; as well as The video decoder according to any one of claims 1-12 or the video processing apparatus according to any one of claims 13-15.

Images