Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the field of deep learning. The data processing method is applied to a second processor, the second processor is provided with a DVPP, and the method can comprise the steps that the second processor receives image data forwarded by a first processor; after receiving the image data, the second processor calls the DVPP to pre-process the image data, and trains a neural network model according to the pre-processed image data to obtain an execution result of the to-be-processed task; according to the data processing method, the CPU performance in a data preprocessing scene can be improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning, and in particular to data processing methods and devices. Background Art

[0002] In the field of deep learning, data processing is extremely important for improving the generalization of the network. Before the data enters the training or reasoning of the model, the data needs to go through a series of preprocessing processes. For example, the image needs to be transformed in size, direction, color, etc. to improve the data quality and ultimately improve the model quality. In order for the model to better learn the generalization of different scenarios, more high-quality data and more precisely designed networks are usually required. However, because massive data is expensive and it is difficult for the data collected from actual scenarios to 100% cover the scenarios that may be encountered during the model reasoning process, the industry generally enhances the training data to generalize the model at a lower cost. Data enhancement has become an important part of the data preprocessing stage. Data preprocessing includes data decoding, data enhancement, etc. With the increase of massive data, the computational overhead of this part even exceeds the subsequent network training part in the time consumption of the whole network training.

[0003] Currently, the central processing unit (CPU) is used for data preprocessing and network training, which occupies a large amount of CPU computing resources and leads to a decrease in CPU performance. Summary of the invention

[0004] The present application provides a data processing method and device, which can improve CPU performance in data preprocessing scenarios.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a data processing method is provided, which is applied to a second processor, wherein the second processor is provided with a digital vision preprocessor DVPP. The method may include: the second processor receives image data forwarded by the first processor, and calls DVPP to preprocess the image data, and then trains a neural network model based on the preprocessed image data to obtain an execution result of the task to be processed.

[0007] In the first aspect, the image data is preprocessed by the DVPP in the second processor, and the neural network model is trained based on the preprocessed image data, and finally the execution result of the task to be processed is obtained. The DVPP has a strong image data processing capability and can efficiently preprocess data, thereby improving the network training efficiency. It also effectively reduces the computing power resource usage of the first processor and ensures the performance of the first processor.

[0008] In a possible implementation, the second processor receiving the image data may include: the second processor calling an operator interface of the DVPP to receive the image data.

[0009] In this implementation, the second processor calls the operator interface of the DVPP to receive the image data, so that the second processor can efficiently obtain the image data.

[0010] In one possible implementation, the digital vision preprocessor includes a vision preprocessing node VPC; calling DVPP to preprocess image data may include: calling VPC in DVPP to convert image data to obtain readable data that can be recognized by a neural network model, and the readable data is the preprocessed image data.

[0011] In this implementation, the image data is converted through VPC, which has a built-in dedicated algorithm for efficiently converting image data, thereby achieving efficient conversion of image data.

[0012] In a possible implementation, the VPC is provided with an image content modification interface and an image format conversion interface; calling the VPC in the DVPP to convert the image data to obtain readable data that can be recognized by the neural network model may include:

[0013] The image content modification interface of the VPC is called to modify the image data into an image that complies with preset rules that define the size, direction, and color of the image, and then the image format conversion interface of the VPC is called to convert the image that complies with the preset rules into readable data.

[0014] In this implementation, image data is converted into readable data for the neural network through a unique interface deployed in the VPC, achieving efficient preprocessing of image data.

[0015] In a possible implementation, the VPC is further provided with a memory transfer interface, and the method may further include: calling the memory transfer interface to store the readable data in a mainboard memory of an electronic device provided with the first processor and the second processor, and releasing the cached data in the DVPP.

[0016] In this implementation, the cached data in the DVPP can be released, thereby improving the DVPP performance.

[0017] In a possible implementation, the second processor includes a network processor and / or a graphics processor.

[0018] In this implementation, by executing the method described in the first aspect and each possible implementation of the first aspect through the second processor (network processor or graphics processor), the computing power resource usage of the first processor can be effectively reduced and the performance of the central processing unit can be guaranteed.

[0019] In a second aspect, the present application provides a data processing device, which is applied to a second processor, and the second processor is provided with a digital vision preprocessor DVPP. The data processing device can be a second processor or a chip or a system on chip in the second processor. The data processing device can implement the functions performed by the second processor in the above-mentioned first aspect or a possible design of the first aspect, and the functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. For example, the data processing device includes: a transceiver module for receiving image data forwarded by the first processor; a processing module for calling DVPP to preprocess the image data; and a processing module for training a neural network model based on the preprocessed image data to obtain the execution result of the task to be processed.

[0020] In a possible implementation, the transceiver module is specifically used to call the operator interface of the DVPP to receive the image data.

[0021] In one possible implementation, the digital vision preprocessor includes a vision preprocessing node VPC; a processing module, specifically used to call the VPC in the DVPP to convert image data to obtain readable data that can be recognized by the neural network model, and the readable data is the preprocessed image data.

[0022] In a possible implementation, the VPC is provided with an image content modification interface and an image format conversion interface; the processing module is specifically used to call the image content modification interface of the VPC to modify the image data into an image that complies with preset rules; wherein the preset rules define the size, direction and color of the image; and the image format conversion interface of the VPC is called to convert the image that complies with the preset rules into readable data.

[0023] In a possible implementation, the VPC is further provided with a memory transfer interface, and the processing module is further used to: call the memory transfer interface to store the readable data in the motherboard memory of the electronic device provided with the first processor and the second processor, and release the cached data in the DVPP.

[0024] In a possible implementation, the second processor includes a network processor and / or a graphics processor.

[0025] In a third aspect, the present application provides a chip, comprising: at least one processor and at least one memory, wherein the at least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, and when the instructions are executed by the at least one processor, the chip executes the first aspect and each possible implementation method of the first aspect.

[0026] In a fourth aspect, the present application provides an electronic device, the electronic device comprising a first processor and a second processor, the first processor being used to obtain image data of a task to be processed and to send the image data to the second processor; the second processor being used to support the electronic device in executing the first aspect and each possible implementation method of the first aspect. Further, the electronic device may also include a memory, the memory storing computer instructions, and the processor may execute the computer instructions to execute the first aspect and each possible implementation method of the first aspect.

[0027] In a fifth aspect, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed, the first aspect and the methods that may be implemented in the first aspect are executed.

[0028] In a sixth aspect, the present application provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer can execute the above-mentioned first aspect and the methods that can be implemented in the first aspect.

[0029] Among them, the beneficial effects described in the second to sixth aspects of the present application can refer to the analysis of the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the structure of a digital vision preprocessor provided in an embodiment of the present application;

[0031] Figure 2 A software architecture block diagram of a digital vision preprocessor provided in an embodiment of the present application;

[0032] Figure 3 A software architecture block diagram of another digital vision preprocessor provided in an embodiment of the present application;

[0033] Figure 4 A software architecture block diagram of another digital vision preprocessor provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 6 A flowchart of a data processing method provided in an embodiment of the present application;

[0036] Figure 7 A flowchart of another data processing method provided in an embodiment of the present application;

[0037] Figure 8 A flowchart of another data processing method provided in an embodiment of the present application;

[0038] Fig. 9 A flowchart of another data processing method provided in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of the time consumption of a CPU preprocessing image data provided in an embodiment of the present application;

[0040] Fig.11 A schematic diagram of the time consumption of preprocessing image data by a digital vision preprocessor provided in an embodiment of the present application;

[0041] Fig.12 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0042] Fig.13 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0044] Before introducing the embodiments of the present application, some terms involved in the embodiments of the present application are explained.

[0045] Deep learning: A type of machine learning technology based on multi-level neural network algorithms. Its main feature is to use multiple nonlinear transformation structures to process and analyze massive data, thereby learning the generalization laws of data. It is mainly used in the field of artificial intelligence, such as machine vision, speech, and natural language processing.

[0046] Data preprocessing (pre-processing): The preprocessing process of massive data in deep learning. This process performs necessary processing on various types of data used for deep learning training, and passes the processed data into the deep learning network. In order for the network to better learn the generalization of the data, relevant preprocessing operations must be performed. Such as image decoding and image enhancement in machine vision scenarios. In the embodiment of the present application, the function of data preprocessing is expanded, in addition to image decoding and image enhancement, it also includes: adjustment of image size, direction and color, and conversion of image format. The preprocessed image in the embodiment of the present application can be directly read by the neural network model, which improves the image processing efficiency.

[0047] CPU: General-purpose processor. It is used in various computing paradigms and has the most general computing capabilities, but does not have the acceleration capabilities of specific computing paradigms. In the deep learning process, the CPU undertakes many heavy tasks other than the accelerated computing of dedicated accelerators, such as data reading, deep learning framework operator compilation, operator distribution, etc. Operators that cannot be accelerated by dedicated accelerators will also be downgraded to be processed by the CPU. Therefore, in deep learning practice, in order to avoid system-level jams, it is necessary to control the CPU workload and reduce the CPU occupancy rate.

[0048] Graphics processing unit (GPU): An accelerator for image processing related tasks. To accelerate image processing, GPUs are equipped with massively parallel processing units, namely CUDA Cores, which also include image-specific processing units such as Texture Units. At the same time, in order to accelerate matrix operations in deep learning, new products have added matrix operation units, Tensor Cores. Overall, GPUs started early in the field of deep learning and have the most mature ecosystem. However, because they evolved from graphics processors, they need to take into account a large number of non-deep learning computing scenarios, so they cannot achieve extreme acceleration for the deep learning paradigm.

[0049] Neural Processing Unit (NPU): A type of domain-specific computing accelerator that focuses on accelerating neural network computing in the deep learning process. It accelerates neural network computing paradigms such as matrix multiplication, convolution, activation function, gradient transfer, etc. Unlike general purpose graph processing units (GPGPUs), NPUs are extremely optimized for the neural network computing paradigm of deep learning, and their computing units are more specialized.

[0050] In the field of deep learning, data processing is extremely important for improving the generalization of the network. Before data enters the training or reasoning of the model, the data needs to go through a series of preprocessing processes to meet the application requirements of the model in order to obtain training data. However, because massive data is expensive and the data collected from actual scenarios is difficult to 100% cover the scenarios that may be encountered during the model reasoning process, the industry generally enhances the training data to generalize the model at a lower cost. Data enhancement has become an important part of the data preprocessing stage.

[0051] Currently, data preprocessing and network training are performed by the CPU. As a general-purpose computing unit, the CPU does not have dedicated supporting resources for efficient data preprocessing, and cannot fully accelerate data preprocessing. The data preprocessing efficiency is low, resulting in low network training efficiency.

[0052] Moreover, with the growth of massive data, the computational overhead of data decoding, data enhancement and other processing processes in data preprocessing is increasing, occupying more CPU computing resources and resulting in a decrease in CPU performance.

[0053] In order to solve the above technical problems, an embodiment of the present application provides a data processing method, which can efficiently perform data preprocessing, ensure network training efficiency, and effectively reduce the computing power resource usage of the CPU, thereby ensuring the performance of the CPU.

[0054] The execution subject of the data processing method provided in the embodiment of the present application can be an electronic device with data processing capabilities such as a computer or a server, or a functional module in the electronic device. Among them, the server can be a single server, or it can be a server cluster composed of multiple servers. In some implementations, the server cluster can also be a distributed cluster. The present application does not limit the execution subject of the data processing method.

[0055] The method provided in the embodiment of the present application is described below in conjunction with the accompanying drawings.

[0056] Below Figure 1 Taking the structure of the digital vision preprocessing processor (DVPP) shown in FIG. 1 as an example, the DVPP provided in the embodiment of the present application is introduced.

[0057] DVPP consists of six modules, namely video decoding (VDEC) module, video encoding (VENC) module, JPEG decoding (JPEGD) module, JPEG encoding (JPEGE) module, PNG decoding (PNGD) module and visual preprocessing (VPC) module.

[0058] The VDEC module provides H.264 / H.265 video decoding function, decodes the input video stream and outputs images for pre-processing in scenarios such as video recognition.

[0059] The VNEC module provides the encoding function for output videos. For the output data of the visual preprocessing module or the original input YUV format data, the video encoding module encodes and outputs H.264 / H.265 videos, which is convenient for direct video playback and display.

[0060] The JPEGD module decodes JPEG format images and converts the original input JPEG images into YUV data.

[0061] The JPEGE encoding module restores the data processed by the JPEGD module to JPEG format for post-processing of the neural network's inference output data.

[0062] When the input image format is PNG, the PNGD decoding module needs to be called for decoding, and the PNG image is output in RGB format for network training.

[0063] The VPC module provides other processing functions for images and videos, such as format conversion (for example, converting YUV format to RGB format, resizing, cropping, etc. In the embodiment of the present application, considering that the training data of machine vision usually comes from video frame extraction or camera acquisition, YUV color encoding is used to be close to human visual perception, and JPEG format compression storage is used to reduce storage and transmission overhead. When the computer processes the image, it needs to use RGB color mode encoding to process the three primary colors, and use bitmap BITMAP format for pixel-by-pixel data processing. In the embodiment of the present application, an interface that can adjust the image color mode, image size, direction and format is set in the VPC module to realize the conversion of JPEG&YUV images into BITMAP&RGB images.

[0064] The interfaces provided in the VPC module provided in the embodiment of the present application include two categories: an image content modification interface and an image format conversion interface. Among them, the image content modification interface includes a color mode conversion interface, an image scaling interface Resize, an image flipping interface Flip, and an image cropping interface Clip. Specifically, the color mode conversion interface can modify the color space of the image data to an image in RGB color mode. The image scaling interface Resize and the image cropping interface Clip can crop and scale the image so that the image can meet the readability requirements of the neural network model. The image flipping interface Flip can adjust the angle of the image to obtain a new image to enrich the training data of the neural network model.

[0065] The image format conversion interface includes the normalization interface Normalize and the image format modification interface Cast. Specifically, the normalization interface Normalize can eliminate the dimensionality effect of image data, make the data more comparable, and eliminate the impact of uneven feature importance on model results, thereby improving the convergence speed of network training. The image format modification interface Cast can modify the format of image data to BITMAP, so that the image data can be read by the neural network model.

[0066] Figure 1 The structure of the DVPP shown is only an example provided in this embodiment. Depending on the needs of the user, the DVPP may also include more or fewer functional modules, which is not limited in this application. For example, in one embodiment, the VPC module may also be provided with a memory transfer interface DMA, which can store the data stored in the DVPP into the motherboard memory of the electronic device, and then release the cached data in the DVPP.

[0067] The present application also provides a software architecture of DVPP, such as Figure 2 As shown, Figure 2 The software architecture block diagram of the DVPP provided in the embodiment of the present application is shown. In the embodiment of the present application, the software architecture of the DVPP is divided into a deep learning framework layer, a request response layer, and a data processing layer. Among them, the deep learning framework layer is used to start multiple concurrent processes to call the request response layer through the operator interface opened by DVPP (the operator interface can be integrated into the mainstream deep learning framework through the Adapter / Plugin method) to obtain a data preprocessing request, and then call the data processing layer to transfer the image data of the task to be processed from the CPU side to the DVPP for preprocessing, and then send the preprocessed data through the queue to the processing core in the NPU (or the processing core in the GPU) for network computing.

[0068] For example, Figure 3 As shown, in the software architecture of DVPP, the deep learning framework layer can adopt the Pytorch framework, and the request response layer in the Pytorch framework is the PyTorch Adapter.

[0069] In another example, if Figure 4 As shown in the figure, in the software architecture of DVPP, the deep learning framework layer can also use the TF framework, and the request response layer in the TF framework is the TF Plugin. In this example, the data processing layer also has intermediate representation (IR) generation components unique to the TF framework: IR_DVPP and IR_NN, which can represent the structure and calculation process of the neural network model in the form of a computational graph.

[0070] The description of the software architecture of DVPP in the embodiment of the present application is an exemplary introduction. In the specific implementation, other software architectures can also be used, and the embodiment of the present application is not limited.

[0071] The hardware structure and software architecture of the DVPP provided in the embodiment of the present application are introduced above. Based on the DVPP introduced above, the embodiment of the present application provides a data processing method, which can be applied to a second processor in an electronic device. The structural diagram of the electronic device is shown in FIG. Figure 5As shown, the electronic device is provided with a first processor and a second processor, and a DVPP is provided in the second processor. Among them, the first processor is a general-purpose processor, which is used for various computing paradigms, has the most general computing power, and does not have the acceleration capability of a specific computing paradigm. In the process of deep learning, the first processor undertakes many heavy tasks other than the dedicated accelerator to accelerate the calculation, such as data reading, deep learning framework operator compilation, operator distribution, etc. The second processor is a dedicated accelerator with the function of efficiently processing specific computing tasks. For example, the second processor can be an accelerator for image processing related work. Alternatively, the second processor can also be an accelerator that focuses on accelerating neural network calculations in the process of deep learning, and accelerates paradigms of neural network calculations, such as matrix multiplication, convolution, activation function, gradient transfer, and other paradigms.

[0072] The first processor and the second processor can be integrated in a physical device. In this case, the electronic device can be a mobile phone, a tablet computer, or a computer with wireless transceiver function, or a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in unmanned driving, or a vehicle-mounted terminal, etc. The first processor and the second processor can also be set in different physical devices. The physical device where the first processor is located is provided with one or more communication interfaces (such as PCIe interface), and the second processor is encapsulated in a pluggable PCIe device. The PCIe device can be connected to the physical device where the first processor is located through the PCIe interface provided by the first processor, so that the first processor and the second processor can transmit data to realize the data processing method provided in the embodiment of the present application.

[0073] Figure 6 FIG. 1 is a flow chart showing a data processing method provided by an embodiment of the present application. Figure 6 As shown, the method may include the following steps:

[0074] S610: The first processor obtains image data of a task to be processed and sends the image data to the second processor. Correspondingly, the second processor receives the image data.

[0075] Among them, reference Figure 5In the description of the electronic device shown, the first processor may be a CPU, the second processor may be an NPU or a GPU, and the second processor may also use an NPU and a GPU in combination, that is, some of the method flows provided in the embodiments of the present application may be executed by the NPU, and the remaining method flows may be executed by the GPU, and which method flows are processed by the NPU and the GPU respectively can be flexibly set without restriction. The first processor is communicatively connected to the second processor and can transmit data to each other. A neural network model is deployed in the processing core of the second processor, and the neural network model can be used to process image data. Exemplarily, the neural network model can be selected as a convolutional neural network, a recurrent neural network, and a generative adversarial network, etc., which are not limited in the embodiments of the present application. In the embodiments of the present application, the NPU and the GPU are only an optional example of the second processor. In the specific implementation, the second processor can use any one or more processors that can set the DVPP, which are not limited in the embodiments of the present application.

[0076] The second processor can call the operator interface of DVPP to receive image data. Figure 3 or Figure 4 The software architecture of DVPP is introduced, and the operator interface of DVPP can receive the image data from the first processor.

[0077] S620: The second processor calls DVPP to preprocess the image data to obtain preprocessed image data.

[0078] Among them, referring to the introduction of DVPP in the embodiment of the present application, DVPP can process image data into data that meets the readable requirements of the neural network model. The digital vision preprocessor includes a visual preprocessing node VPC, and the second processor can call the VPC in the DVPP to convert the image data into readable data of the neural network model to realize the preprocessing of the image data.

[0079] S630: The second processor trains a neural network model according to the preprocessed image data to obtain an execution result of the task to be processed.

[0080] After the image data is preprocessed by DVPP, the preprocessed image data can be read by the neural network model. At this time, the second processor can train the neural network model according to the preprocessed image data to obtain the execution result of the task to be processed.

[0081] The second processor can call the processing core to train the neural network model. Exemplarily, when the second processor is an NPU, the processing core is an artificial intelligence processing core (AIC). In another example, when the second processor is a GPU, the processing core can be a Cuda Core or a Tensor Core.

[0082] The neural network model can adopt various types of neural networks, for example, the ResNet50 model in the convolutional neural network. The training process of the neural network model is relatively common, and you can refer to the introduction of related technologies, which will not be repeated here.

[0083] In the embodiment of the present application, the image data of the task to be processed is obtained by the first processor, and then the DVPP in the second processor pre-processes the image data, and the neural network model is trained based on the pre-processed image data, and finally the execution result of the task to be processed is obtained. The image data processing capability of DVPP is strong, and it can efficiently pre-process data, thereby improving the efficiency of network training. It also effectively reduces the computing power resource usage of the first processor, ensuring the performance of the first processor.

[0084] In one embodiment, the VPC is provided with an image content modification interface and an image format conversion interface. Figure 7 As shown, S620 (calling DVPP to pre-process the image data) may include:

[0085] S710: The second processor calls the image content modification interface of the VPC to modify the image data into an image that complies with a preset rule.

[0086] The preset rules define the size, direction and color of the image. Referring to the introduction of DVPP in the embodiment of the present application, the image content modification interface can adjust the size, direction and color mode of the image, and can modify the image data to an image that complies with the preset rules. For example, Figure 8 As shown, the image content modification interface includes an image cropping interface, an image scaling interface, a color mode conversion interface, and an image flipping interface. The process of modifying image data to an image that conforms to a preset rule can be as follows:

[0087] The second processor calls the color mode conversion interface to modify the resized image data into the target color mode (RGB) image data, then calls the image cropping interface and the image scaling interface to convert the image data into an image of the target size (which can be preset), and then calls the image flipping interface to flip the image data; finally, the image data before and after flipping are determined as images that meet the preset rules. The order of cropping, scaling, and flipping the image data can be flexibly adjusted, and the embodiments of the present application are not limited thereto.

[0088] S720: The second processor calls the VPC in the DVPP to convert the image data to obtain readable data that can be recognized by the neural network model.

[0089] The readable data is pre-processed image data. Referring to the introduction of DVPP in the embodiment of the present application, the image format conversion interface can convert the data into readable data of the neural network model. For example, Fig. 9 As shown, the image format conversion interface includes a normalization interface and an image format modification interface. The process of converting an image that conforms to preset rules into readable data can be as follows:

[0090] The second processor calls the normalization interface to perform normalization processing on the image that meets the preset rules to obtain normalized data, and then calls the image format conversion interface to convert the normalized data into the readable data (in the BITMAP format).

[0091] The VPC may also be provided with a memory transfer interface, which can store readable data into the motherboard memory of the electronic device and release cached data in the DVPP to improve the DVPP performance.

[0092] In the embodiment of the present application, image data is converted into readable data of a neural network through a unique interface deployed in the VPC, thereby achieving efficient preprocessing of the image data.

[0093] In one embodiment, image data is randomly acquired as image data of the task to be processed for a control test. Fig.10 As shown, the randomly acquired image data is preprocessed based on the CPU: decoding takes 2.5ms, cropping takes 1.4ms, flipping takes 0.09ms, tensor conversion takes 0.28ms, normalization takes 0.19ms, and a total of 4.46ms.

[0094] like Fig.11 As shown, the method provided in the embodiment of the present application is used for data preprocessing, and specifically, the GPU's DVPP is used to preprocess the randomly acquired image data. Among them, it takes 460us for the CPU to send processing instructions to the GPU, 250us for the CPU and GPU to synchronize data, 400us for DVPP decoding, 150us for DVPP cropping, 50us for DVPP flipping, 100us for DVPP tensor conversion, and 100us for DVPP normalization, which takes a total of about 1.1ms.

[0095] By comparing the above Fig.10 and Fig.11It can be seen from the test time consumption shown respectively that the data preprocessing efficiency of DVPP in the embodiment of the present application is significantly better than the efficiency of CPU in data preprocessing.

[0096] In the embodiment of the present application, the image data of the task to be processed is obtained by the first processor, and then the DVPP in the second processor pre-processes the image data, and the neural network model is trained based on the pre-processed image data, and finally the execution result of the task to be processed is obtained. The image data processing capability of DVPP is strong, and it can efficiently pre-process data, thereby improving the efficiency of network training. It also effectively reduces the computing power resource usage of the first processor, ensuring the performance of the first processor.

[0097] The above mainly introduces the scheme provided by the embodiment of the present application from the perspective of the execution logic of each step. It is understandable that each node, such as the second processor, includes a hardware structure and / or software module corresponding to each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiment disclosed herein, the method of the embodiment of the present application can be implemented in the form of hardware, software, or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0098] The embodiment of the present application can divide the functional modules of the second processor according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0099] Fig.12 A structural diagram of a data processing device 1200 is shown. The data processing device is applied to a second processor in an electronic device provided with a first processor and a second processor, and the second processor is provided with a digital vision preprocessor DVPP. Fig.12 Each module in the device shown has the function of realizing Figure 1-Figure 11 The functions of the corresponding steps in the module can achieve the corresponding technical effects. The corresponding beneficial effects of each module execution step can be referred to Figure 1-Figure 11The description of the corresponding steps is not repeated here. The functions can be implemented by hardware or by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above functions. The data processing device can be a second processor or a chip or system on chip in the second processor. For example, the data processing device includes:

[0100] The transceiver module 121 receives the image data forwarded by the first processor; the processing module 122 is used to call DVPP to preprocess the image data; the processing module 122 is also used to train the neural network model according to the preprocessed image data to obtain the execution result of the task to be processed.

[0101] In one embodiment, the transceiver module 121 is specifically configured to call an operator interface of the DVPP to receive image data.

[0102] In one embodiment, the digital vision preprocessor includes a vision preprocessing node VPC; a processing module 122, which is specifically used to call the VPC in the DVPP to convert image data to obtain readable data that can be recognized by the neural network model, and the readable data is the preprocessed image data.

[0103] In one embodiment, the VPC is provided with an image content modification interface and an image format conversion interface; the processing module 122 is specifically used to call the image content modification interface of the VPC to modify the image data into an image that complies with preset rules; wherein the preset rules define the size, direction and color of the image; and call the image format conversion interface of the VPC to convert the image that complies with the preset rules into readable data.

[0104] In one embodiment, the VPC is further provided with a memory transfer interface, and the processing module 122 is further used to: call the memory transfer interface to store the readable data into the motherboard memory of the electronic device, and release the cached data in the DVPP.

[0105] In one embodiment, the second processor includes a network processor and / or a graphics processor.

[0106] The present application also provides an electronic device in an embodiment, the electronic device includes a first processor and a second processor, the first processor is used to obtain image data of a task to be processed and send the image data to the second processor; the second processor is used to support the electronic device to execute all or part of the processes in the above method embodiment. Further, the electronic device may also include a memory, the memory stores computer instructions, and the first processor and the second processor can run the computer instructions to execute the data processing method described in the embodiment of the present application.

[0107] In specific implementation, the electronic device shown in this application can be used Fig.13 The structure shown may include Fig.13 Parts shown. Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. When the electronic device has the functions of the electronic device described in the embodiment of the present application, the electronic device may be an electronic device or a chip or system on chip in the electronic device.

[0108] like Fig.13 As shown, the electronic device may include a processor 901, a communication line 902, a transceiver 903, a memory 904, and a processor 907. The processor 901, the processor 907, the memory 904, and the transceiver 903 may be connected via the communication line 902.

[0109] The processor 901 or the processor 907 may be a CPU, a GPU, an NPU or any combination thereof. The processor 901 or the processor 907 may also be other devices with processing functions, such as circuits, devices or software modules.

[0110] The communication line 902 is used to transmit information between the components included in the electronic device.

[0111] The transceiver 903 is used to communicate with other devices or other communication networks. The other communication networks may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The transceiver 903 may be an interface circuit, a pin, a radio frequency module, a transceiver or any device capable of achieving communication.

[0112] Furthermore, the electronic device may also include a memory 904. The memory 904 is used to store instructions, wherein the instructions may be computer programs.

[0113] Among them, the memory 904 can be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage, magnetic disk storage medium or other magnetic storage device, and the optical disc storage includes a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, or a Blu-ray disc, etc.

[0114] It should be noted that the memory 904 can exist independently of the processor 901, or can be integrated with the processor 901. The memory 904 can be used to store instructions or program codes or some data, etc. The memory 904 can be located inside the electronic device or outside the electronic device, without limitation. When the processor 901 executes the instructions stored in the memory 904, the method provided in the embodiment of the present application can be implemented.

[0115] As an optional implementation, the electronic device further includes an output device 905 and an input device 906. Exemplarily, the input device 906 is a keyboard, a mouse, a microphone, a joystick, and the like, and the output device 905 is a display screen, a speaker, and the like.

[0116] It should be noted that electronic devices can be desktop computers, portable computers, network servers, mobile phones, tablet computers, wireless terminals, embedded devices, chip systems or Fig.13 In addition, Fig.13 The components shown in the figure do not constitute a limitation on the electronic device, except Fig.13 In addition to the components shown, the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0117] In the embodiment of the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices.

[0118] The embodiment of the present application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be completed by a computer program to instruct the relevant hardware, and the program can be stored in the above computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be a terminal device of any of the above embodiments, such as: an internal storage unit including a data sending end and / or a data receiving end, such as a hard disk or memory of a terminal device. The above computer-readable storage medium can also be an external storage device of the above terminal device, such as a plug-in hard disk equipped on the above terminal device, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. Further, the above computer-readable storage medium can also include both an internal storage unit of the above terminal device and an external storage device. The above computer-readable storage medium is used to store the above computer program and other programs and data required by the above terminal device. The above computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0119] The present application also provides a computer instruction. All or part of the process in the above method embodiment can be completed by computer instructions to instruct related hardware (such as computers, processors, network devices, and terminals, etc.). The program can be stored in the above computer-readable storage medium.

[0120] The embodiment of the present application also provides a chip. The chip may be composed of a chip, or may include a chip and other discrete devices, without limitation. The chip includes at least one processor and at least one memory, the at least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, and when the instructions are executed by the at least one processor, the chip executes all or part of the process in the above method embodiment.

[0121] In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0122] In the embodiment of the present application, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application may also be a circuit or any other device that can implement a storage function, for storing instructions and / or data.

[0123] It should be noted that the terms "first" and "second" in the specification, claims and drawings of the present application are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0124] It should be understood that in the embodiments of the present application, "at least one (item)" refers to one or more, "multiple" refers to two or more, "at least two (items)" refers to two or three and more than three, and "and / or" is used to describe the association relationship of the associated objects, indicating that there can be three relationships. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following (items)" or similar expressions refers to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A. For example, B can be determined based on A. It should also be understood that determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. In addition, the "connection" in the embodiments of the present application refers to various connection modes such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitation on this.

[0125] Unless otherwise specified, the "transmission" (transmit / transmission) appearing in the embodiments of the present application refers to bidirectional transmission, including sending and / or receiving actions. Specifically, the "transmission" in the embodiments of the present application includes the sending of data, the receiving of data, or the sending of data and the receiving of data. In other words, the data transmission here includes uplink and / or downlink data transmission. Data may include channels and / or signals, uplink data transmission is uplink channel and / or uplink signal transmission, and downlink data transmission is downlink channel and / or downlink signal transmission. The "network" and "system" appearing in the embodiments of the present application express the same concept, and the communication system is the communication network.

[0126] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0127] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0129] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device, such as: a single-chip microcomputer, a chip, etc., or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, disks, or optical disks.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method is applied to a second processor, wherein the second processor includes a digital vision preprocessor DVPP, and the method includes: receiving image data of a task to be processed, wherein the image data is forwarded by the first processor to the second processor; Calling DVPP to pre-process the image data; A neural network model is trained according to the preprocessed image data to obtain the execution result of the task to be processed.

2. The data processing method according to claim 1, characterized in that: Receiving the image data, comprising: The operator interface of DVPP is called to receive the image data.

3. The data processing method according to claim 1 or 2, characterized in that: The digital vision preprocessor includes a vision preprocessing node VPC; The calling of DVPP to pre-process the image data includes: The VPC in the DVPP is called to convert the image data to obtain readable data that can be recognized by the neural network model, and the readable data is the preprocessed image data.

4. The data processing method according to claim 3, characterized in that: VPC is equipped with an image content modification interface and an image format conversion interface; Calling the VPC in the DVPP to convert the image data to obtain readable data that can be recognized by the neural network model includes: Calling the image content modification interface of the VPC to modify the image data into an image that complies with preset rules; wherein the preset rules define the size, direction, and color of the image; The image format conversion interface of the VPC is called to convert the image that complies with the preset rule into the readable data.

5. The data processing method according to claim 4, characterized in that: The VPC is also provided with a memory transfer interface, and the method further includes: The memory transfer interface is called to store the readable data in a mainboard memory of an electronic device, and the cached data in the DVPP is released, wherein the electronic device includes the first processor and the second processor.

6. The data processing method according to any one of claims 1 to 5, characterized in that: The second processor includes a network processor and / or a graphics processor.

7. A data processing device, characterized in that: The device is applied to a second processor, the second processor includes a digital vision preprocessor DVPP, and the device includes: A transceiver module, used for receiving image data of a task to be processed, wherein the image data is forwarded from the first processor to the second processor; A processing module, used for calling DVPP to pre-process the image data; The processing module is also used to train a neural network model based on the preprocessed image data to obtain the execution result of the task to be processed.

8. The data processing device according to claim 7, characterized in that: The transceiver module is specifically used for: The operator interface of DVPP is called to receive the image data.

9. The data processing device according to claim 7 or 8, characterized in that: The digital vision preprocessor comprises a vision preprocessing node VPC; the processing module is specifically used for: The VPC in the DVPP is called to convert the image data to obtain readable data that can be recognized by the neural network model, and the readable data is the preprocessed image data.

10. The data processing device according to claim 9, characterized in that: The VPC is provided with an image content modification interface and an image format conversion interface; the processing module is specifically used for: Calling the image content modification interface of the VPC to modify the image data into an image that complies with preset rules; wherein the preset rules define the size, direction, and color of the image; The image format conversion interface of the VPC is called to convert the image that complies with the preset rule into the readable data.

11. The data processing device according to claim 10, characterized in that: The VPC is also provided with a memory transfer interface, and the processing module is further used for: The memory transfer interface is called to store the readable data in a mainboard memory of an electronic device, and the cached data in the DVPP is released, wherein the electronic device includes the first processor and the second processor.

12. The data processing device according to any one of claims 7 to 11, characterized in that: The second processor includes a network processor and / or a graphics processor.

13. A chip, characterized in that: The chip comprises: at least one processor; At least one memory, the at least one memory is coupled to the at least one processor element and stores instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the chip to perform the method as claimed in any one of claims 1 to 6.

14. An electronic device, characterized in that: The electronic device comprises a first processor and a second processor, wherein the first processor is used to obtain image data of a task to be processed and send the image data to the second processor; The second processor is used to support the electronic device to execute the method according to any one of claims 1-6.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed, the method according to any one of claims 1 to 6 is executed.

Images