A data processing method and apparatus

Abstract

A data processing method applied to image processing, such as image resampling, the method comprising: obtaining a first image, the first image comprising a plurality of first pixel points; obtaining, according to the first image, a distribution parameter corresponding to each first pixel point through a target mapping relationship (such as a neural network or a mapping relationship obtained based on a neural network), the distribution parameter being related to a pixel value distribution of pixel points around the first pixel point; determining a weight corresponding to each first pixel point according to a resampling function corresponding to each first pixel point in the plurality of first pixel points, the weight being used to fuse pixel values of the plurality of first pixel points into a pixel value corresponding to a target pixel point; and constructing a resampling image corresponding to the first image according to the pixel value corresponding to the target pixel point. The present application can determine the distribution parameter of each pixel point through the neural network and then determine the resampling function, so that different pixels can correspond to different resampling functions, thereby improving the quality of the resampling image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method and device thereof. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0003] Image resampling is a major research direction of underlying computer vision. In recent years, methods based on convolutional neural network (CNN) have achieved remarkable results in image resampling performance. For CNN methods, generally the more complex the network, the better the results. Complex structures and relatively large parameters can better learn the characteristics of the resampling problem under different geometric transformations from data.

[0004] In the prior art, when performing image resampling, a coordinate bias can be brought into a fixed resampling function to obtain a weight value of a local pixel in a current coordinate system corresponding to each pixel in an output picture.

[0005] However, in the prior art, different interpolation methods all use fixed resampling functions, such as nearest neighbor function, linear function, cubic function, etc. The calculated weight is only related to the coordinate bias and is irrelevant to the input picture. Therefore, the interpolation method is difficult to preserve the unique texture details of different pictures, and the accuracy of image resampling is poor. SUMMARY

[0006] The present application provides a data processing method, which can improve the processing accuracy of image resampling.

[0007] In a first aspect, the present application provides an image processing method, comprising: obtaining a first image, the first image comprising a plurality of first pixel points; obtaining, according to the first image, a distribution parameter corresponding to each of the first pixel points through a target mapping relationship, the distribution parameter being related to a pixel value distribution of pixel points around the first pixel point; wherein the target mapping relationship is a neural network, or the target mapping relationship comprises pixel combination categories of the pixel points around the first pixel point and a distribution parameter corresponding to each of the pixel combination categories, wherein the distribution parameter corresponding to each of the pixel combination categories is an output result obtained by processing a pixel value corresponding to each of the pixel combination categories based on the neural network; the distribution parameter is used to determine a corresponding resampling function; determining a weight corresponding to each of the first pixel points according to the resampling function corresponding to each of the first pixel points, the weight being used to fuse pixel values of the plurality of first pixel points into a pixel value corresponding to a target pixel point; the target pixel point being a pixel point corresponding to the plurality of first pixel points in a resampling image corresponding to the first image; and constructing the resampling image corresponding to the first image according to the pixel value corresponding to the target pixel point.

[0008] In the prior art, the same resampling function is used for each pixel point. In the embodiments of the present application, the distribution parameter of each pixel point can be determined based on an image through a neural network to adjust the resampling function, so that one pixel can correspond to one resampling function, and the distribution parameter is obtained based on the pixel value distribution around the pixel, so that the determined resampling function can more accurately quantify the relationship between pixel bias and weight, and can learn the edge information of the image, thereby improving the quality of the resampling image.

[0009] The pixel combination category can be related to the position of the selected pixel point (such as a pixel point in a square region nearby, or a pixel point in a vertical long strip region), the size of the pixel point, and the arrangement relationship of the pixel point in space. In this case, the pixel values of the plurality of pixel points and the positions (such as absolute positions or relative positions between other pixel points) of the pixel points corresponding to the pixel values in the image are input into the neural network.

[0010] In actual use, for a pixel point, the pixel combination category satisfied by the pixel values of the pixel points around the pixel point can be determined first, and the distribution parameter corresponding to the satisfied pixel combination category is determined from the target mapping relationship.

[0011] In actual use, for a pixel point, a pixel combination type satisfied by pixel values of pixel points around the pixel point can be determined first, and a distribution parameter corresponding to the satisfied pixel combination type is determined from the target mapping relationship. That is, since the input is a limited pixel value combination, the neural network can not be stored, but only the corresponding relationship between the input and the output of the neural network is stored, thereby reducing the storage cost (without the need to store the neural network) and the calculation cost (without the need to perform the inference process of the neural network). Compared with existing interpolation algorithms and CNN-based methods, a better trade-off between performance and power consumption (or performance and running time delay) is achieved, and better image resampling performance can be achieved with smaller power consumption and running time delay.

[0012] In a possible implementation, the resampling function represents a corresponding relationship between coordinate offsets and weights, and the distribution parameter is used to constrain a distribution relationship of the pixel offsets and the weights on at least one of the following information:

[0013] A transverse and longitudinal correlation coefficient, a transverse standard deviation, and a longitudinal standard deviation.

[0014] In a possible implementation, the resampling function is as follows:

[0015]

[0016] wherein the d represents a coordinate offset, the p, s X , s Y are distribution parameters, and the s is a covariance matrix obtained from the distribution parameters.

[0017] In a possible implementation, the weight is used to fuse pixel values of a plurality of first pixel points into a pixel value corresponding to a target pixel point, including that the weight is used to obtain the pixel value corresponding to the target pixel point by weighted average of the pixel values of the plurality of first pixel points.

[0018] In a possible implementation, the distribution parameter corresponding to each first pixel point is obtained from the first image through a target mapping relationship, including that a distribution parameter corresponding to a pixel combination type satisfied by pixel values of a plurality of pixel points located at a preset position near each first pixel point in the first image is obtained through a target mapping relationship.

[0019] In a possible implementation, the distribution parameter corresponding to each first pixel point is obtained from the first image through a target mapping relationship, including:

[0020] According to a first pixel combination type satisfied by pixel values of a plurality of pixel points at a first preset position near each of the first pixel points in the first image, a first distribution parameter corresponding to the first pixel combination type is obtained through a target mapping relationship; according to a second pixel combination type satisfied by pixel values of a plurality of pixel points at a second preset position near each of the first pixel points in the first image, a second distribution parameter corresponding to the second pixel combination type is obtained through the target mapping relationship; and the first distribution parameter and the second distribution parameter are used to obtain the distribution parameter through fusion.

[0021] In this embodiment, the neural network inference is accelerated by using the lookup table acceleration technology, and the edge index and edge aggregation strategy are designed to further extract and perceive the edge information of the local area pixels, so that better power consumption advantage and edge preservation effect are achieved.

[0022] In a second aspect, the present application provides an image processing method, the method comprising:

[0023] obtaining a first image, the first image comprising a plurality of first pixel points;

[0024] According to the first image, a distribution parameter corresponding to each of the first pixel points is obtained through a target mapping relationship, the distribution parameter being related to a pixel value distribution of pixel points around the first pixel point; wherein the target mapping relationship is a neural network, or the target mapping relationship comprises a pixel combination type of the surrounding pixel points and a distribution parameter corresponding to each pixel combination type, wherein the distribution parameter corresponding to each pixel combination type is an output result obtained by processing a pixel value corresponding to each pixel combination type based on a neural network; and the distribution parameter is used to determine a corresponding resampling function;

[0025] According to the resampling function corresponding to each of the first pixel points, a weight corresponding to each of the first pixel points is determined, the weight being used to fuse pixel values of a plurality of first pixel points into a pixel value corresponding to a target pixel point; and the target pixel point is a pixel point corresponding to the plurality of first pixel points in a resampling image corresponding to the first image.

[0026] According to the pixel value corresponding to the target pixel point, a resampling image corresponding to the first image is constructed.

[0027] According to the resampling image and a corresponding true value image, the neural network is updated.

[0028] In a possible implementation, the resampling function represents a corresponding relationship between a coordinate bias and a weight, and the distribution parameter is used to constrain a distribution relationship between the pixel bias and the weight on at least one of the following information:

[0029] The transverse and longitudinal correlation coefficients, the transverse standard deviation, and the longitudinal standard deviation.

[0030] In a possible implementation, the resampling function is as follows:

[0031]

[0032] wherein the d represents a coordinate bias, the p, s X , s Y are distribution parameters, and the S is a covariance matrix derived from the distribution parameters.

[0033] In a possible implementation, the weight is used to fuse pixel values of a plurality of first pixel points into a pixel value corresponding to a target pixel point, including:

[0034] The weight is used to obtain the pixel value corresponding to the target pixel point through weighted average of the pixel values of the plurality of first pixel points.

[0035] In a third aspect, the present application provides a data processing apparatus, including:

[0036] The acquisition module is configured to acquire a first image, the first image including a plurality of first pixel points.

[0037] The processing module is configured to obtain, according to the first image, a distribution parameter corresponding to each first pixel point through a target mapping relationship, the distribution parameter being related to a pixel value distribution of pixel points around the first pixel point; wherein the target mapping relationship is a neural network, or the target mapping relationship includes pixel combination types of the surrounding pixel points and a distribution parameter corresponding to each pixel combination type, wherein the distribution parameter corresponding to each pixel combination type is an output result obtained based on neural network processing of a pixel value corresponding to each pixel combination type; and the distribution parameter is used to determine a corresponding resampling function.

[0038] According to the resampling function corresponding to each first pixel point in the plurality of first pixel points, a weight corresponding to each first pixel point is determined, the weight being used to fuse pixel values of a plurality of first pixel points into a pixel value corresponding to a target pixel point; and the target pixel point is a pixel point corresponding to the plurality of first pixel points in a resampling image corresponding to the first image.

[0039] According to the pixel value corresponding to the target pixel point, a resampling image corresponding to the first image is constructed.

[0040] In a possible implementation, the resampling function represents a corresponding relationship between a coordinate bias and a weight, and the distribution parameter is used to constrain a distribution relationship between the pixel bias and the weight on at least one of the following information:

[0041] The transverse and longitudinal correlation coefficients, the transverse standard deviation, and the longitudinal standard deviation.

[0042] In a possible implementation, the resampling function is as follows:

[0043]

[0044] wherein the d represents a coordinate bias, the p, s X , s Y are distribution parameters, and the S is a covariance matrix derived from the distribution parameters.

[0045] In a possible implementation, the weight is used to fuse pixel values of a plurality of first pixel points into a pixel value corresponding to a target pixel point, including:

[0046] The weight is used to obtain the pixel value corresponding to the target pixel point by weighted average of the pixel values of the plurality of first pixel points.

[0047] In a possible implementation, the processing module is specifically configured to obtain, according to a pixel combination type satisfied by pixel values of a plurality of pixel points at a preset position near each first pixel point in the first image, a distribution parameter corresponding to the satisfied pixel combination type through a target mapping relationship.

[0048] In a possible implementation, the processing module is specifically configured to obtain, according to a first pixel combination type satisfied by pixel values of a plurality of pixel points at a first preset position near each first pixel point in the first image, a first distribution parameter corresponding to the first pixel combination type through a target mapping relationship.

[0049] obtain, according to a second pixel combination type satisfied by pixel values of a plurality of pixel points at a second preset position near each first pixel point in the first image, a second distribution parameter corresponding to the second pixel combination type through a target mapping relationship; and the first distribution parameter and the second distribution parameter are used to fuse to obtain the distribution parameter.

[0050] In a fourth aspect, the present application provides a data processing apparatus, including:

[0051] An acquisition module is configured to acquire a first image, the first image including a plurality of first pixel points.

[0052] The processing module is configured to obtain, according to the first image, a distribution parameter corresponding to each first pixel point through a target mapping relationship, the distribution parameter being related to a pixel value distribution of a pixel point around the first pixel point, wherein the target mapping relationship is a neural network, or the target mapping relationship includes a pixel combination type of the pixel point around the first pixel point and a distribution parameter corresponding to each pixel combination type, wherein the distribution parameter corresponding to each pixel combination type is an output result obtained by processing a pixel value corresponding to each pixel combination type based on the neural network, and the distribution parameter is used to determine a corresponding resampling function.

[0053] According to the resampling function corresponding to each first pixel point in the plurality of first pixel points, a weight corresponding to each first pixel point is determined, and the weight is used to fuse pixel values of the plurality of first pixel points into a pixel value corresponding to a target pixel point; and the target pixel point is a pixel point corresponding to the plurality of first pixel points in a resampling image corresponding to the first image.

[0054] According to the pixel value corresponding to the target pixel point, a resampling image corresponding to the first image is constructed.

[0055] The updating module is configured to update the neural network according to the resampling image and a corresponding true value image.

[0056] In a possible implementation, the resampling function represents a corresponding relationship between a coordinate bias and a weight, and the distribution parameter is used to constrain a distribution relationship between the pixel bias and the weight on at least one of the following information:

[0057] A horizontal and vertical correlation coefficient, a horizontal standard deviation, and a vertical standard deviation.

[0058] In a possible implementation, the resampling function is as follows:

[0059]

[0060] wherein the d represents a coordinate bias, the ρ,σ X ,σ Y is a distribution parameter, and the Σ is a covariance matrix obtained based on the distribution parameter.

[0061] In a possible implementation, the processing module is specifically configured to use the weight to obtain the pixel value corresponding to the target pixel point through weighted average of the pixel values of the plurality of first pixel points.

[0062] In a fifth aspect, an embodiment of the present application provides a data processing apparatus, which can include a memory, a processor and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform the method of the first aspect and any optional method of the first aspect, and the method of the second aspect and any optional method of the second aspect.

[0063] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer program causes the computer to perform the method of the first aspect and any optional method of the first aspect, and the method of the second aspect and any optional method of the second aspect.

[0064] In a seventh aspect, an embodiment of the present application provides a computer program, which, when run on a computer, causes the computer to perform the method of the first aspect and any optional method of the first aspect, and the method of the second aspect and any optional method of the second aspect.

[0065] In an eighth aspect, the present application provides a chip system, which includes a processor configured to support the execution of the data processing apparatus to perform the functions involved in the above aspects, for example, sending or processing the data involved in the above methods; or, information. In a possible design, the chip system further includes a memory, and the memory is configured to store the necessary program instructions and data for the execution device or the training device. The chip system can be composed of a chip, or can include a chip and other discrete devices. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1A A structural schematic diagram of an artificial intelligence subject framework;

[0067] Figure 1B And to Figure 1C A system framework schematic of the present application;

[0068] Figure 1D An optional hardware structure schematic of a terminal;

[0069] Figure 2 A structural schematic diagram of a server;

[0070] Figure 3 A system architecture schematic of the present application;

[0071] Figure 4 A flow of a cloud service;

[0072] Figure 5A And Figure 5B A structural schematic diagram of a network;

[0073] Figure 6 A flow schematic of a data processing method provided by an embodiment of the present application;

[0074] Figures 7 to 10 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0075] Figure 11A and Figure 11B This is a schematic diagram of one architecture of this application;

[0076] Figure 11C This is a schematic diagram of a resampling function according to this application;

[0077] Figure 11D This is a schematic diagram of a edge-preserving aggregation method according to this application;

[0078] Figure 12 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;

[0079] Figure 13 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0080] Figure 14 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0081] Figure 15 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0082] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0083] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0084] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0085] The terms "substantially," "about," and similar terms as used herein are taken to mean approximately, as in a tolerably close or approximate value, and not to mean exactly or perfectly as in a precise value. Additionally, the use of "may" when describing embodiments herein is meant to convey that one or more embodiments of the application can or can not employ such features. The terms "use," "using," and "used" as used herein can be taken in their broadest context as synonymous with the terms "utilize," "utilizing," and "utilized," respectively. In addition, the term "exemplary" is intended to mean an instance or illustration.

[0086] First, the overall workflow of the artificial intelligence system is described, please see Figure 1A , Figure 1A The structure of the artificial intelligence subject framework is shown, and the following describes the artificial intelligence subject framework from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the bottom infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0087] (1) Infrastructure

[0088] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network related platform guarantee and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication acquire data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0089] (2) Data

[0090] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, voice, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, humidity, etc.

[0091] (3) Data processing

[0092] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision, etc.

[0093] Among them, machine learning and deep learning can model, extract, preprocess, train, etc. symbolic and formalized intelligent information on data.

[0094] Reasoning refers to the process of simulating human intelligent reasoning methods in a computer or intelligent system, using formalized information to solve problems according to reasoning control strategies, and the typical function is search and matching.

[0095] Decision refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0096] (4) General capabilities

[0097] After data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0098] (5) Intelligent products and industry applications

[0099] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are encapsulations of overall solutions of artificial intelligence, and realize application landing by productizing intelligent information decision-making. The application fields mainly include intelligent terminals, intelligent transportation, intelligent medical treatment, autonomous driving, smart cities, etc.

[0100] The present application can be applied to the field of image processing in the field of artificial intelligence. The following will introduce multiple application scenarios landed in products by taking image processing as an example.

[0101] First, the application scenario of the present application is introduced.

[0102] The present application can be but is not limited to applied in an application program with image processing function (hereinafter can be referred to as an image processing application program) or a cloud service provided by a cloud-side server, etc. Next, they are introduced respectively:

[0103] I. Image processing application program

[0104] The product form of the present application embodiment can be an image processing application program, in particular, an application program with image resampling function. The image processing application program can run on a terminal device or a server on the cloud side.

[0105] In a possible implementation, the image processing application program can implement image resampling based on the input image, and obtain a processing result, which can be an image resampling result.

[0106] In a possible implementation, the user can open the image processing application program installed on the terminal device, and input an image. The image processing application program can process the image by using the method provided in the embodiments of the present application, and present the processing result to the user (the presentation manner can be, but is not limited to, display, saving, uploading to the cloud side, etc.).

[0107] In a possible implementation, the user can open the image processing application program installed on the terminal device, and input an image. The image processing application program can send the image to the server on the cloud side, the server on the cloud side processes the image by using the method provided in the embodiments of the present application, and returns the processing result to the terminal device. The terminal device can present the processing result to the user (the presentation manner can be, but is not limited to, display, saving, uploading to the cloud side, etc.).

[0108] Next, the image processing application program in the embodiments of the present application is introduced from the functional architecture and the product architecture implementing the function respectively.

[0109] Referring to Figure 1B , Figure 1B , the functional architecture of the image processing application program in the embodiments of the present application is shown as follows:

[0110] In a possible implementation, as shown in Figure 1B , the image processing application program 102 can receive the input parameter 101 (for example, containing an image) and generate a processing result 103. The image processing application program 102 can be executed on, for example, at least one computer system, and includes computer code which, when executed by one or more computers, causes the computers to perform the method provided in the embodiments of the present application.

[0111] Referring to Figure 1C , Figure 1C , the entity architecture of the image processing application program in the embodiments of the present application is shown as follows:

[0112] Referring to Figure 1C , Figure 1C , a system architecture diagram is shown. The system can include a terminal 100 and a server 200. The server 200 can include one or more servers (for example, one server is taken as an example in the description), and the server 200 can provide the method provided in the embodiments of the present application for one or more terminals. Figure 1C

[0113] ​The terminal 100 can install an image processing application. The application and the webpage can provide an interface. The terminal 100 can receive parameters input by a user on the interface, and send the parameters to the server 200. The server 200 can obtain a processing result based on the received parameters, and return the processing result to the terminal 100.

[0114] It should be understood that, in some optional implementations, the terminal 100 can also complete the action of obtaining a processing result based on received parameters by itself without the cooperation of the server. The embodiments of the present application are not limited thereto.

[0115] Next, the product form of the terminal 100 is described. Figure 1C

[0116] The terminal 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like. The embodiments of the present application are not limited thereto.

[0117] Figure 1D An optional hardware structure schematic diagram of the terminal 100 is shown.

[0118] Referring to FIG. 1, Figure 1D As shown in FIG. 1, the terminal 100 can include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, a power supply 190, and the like. Those skilled in the art can understand that, Figure 1D The terminal 100 or the multifunctional device is only an example and does not constitute a limitation on the terminal or the multifunctional device, and can include more or fewer components than those shown, or combine certain components, or different components.

[0119] ​The input unit 130 can be used to receive inputted digital or character information, and to generate key signal inputs related to user settings of the portable multifunctional device and control of functions. Specifically, the input unit 130 can include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect touch operations of a user thereon or therearound (such as operations of the user using a finger, a joint, a stylus, or any suitable object on or near the touch screen), and drive corresponding connected devices according to pre-set programs. The touch screen can detect touch actions of the user on the touch screen, convert the touch actions into touch signals and send the touch signals to the processor 170, and can receive commands from the processor 170 and execute the commands; the touch signals at least include touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, the other input devices 132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc.

[0120] The input device 132 can receive an inputted image, etc.

[0121] The display unit 140 can be used to display information inputted by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playing of any kind of multimedia files. In the embodiments of the present application, the display unit 140 can be used to display interfaces of image processing type application programs, processing results, etc.

[0122] The storage 120 can be used to store instructions and data. The storage 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc.; the storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc., or their subsets, expanded sets. It can also include a non-volatile random access memory; provide the processor 170 with software and applications that include management of hardware, software, and data resources in the computing processing device, support control. It is also used for storage of multimedia files, and storage of running programs and applications.

[0123] The processor 170 is the control center of the terminal 100, connects each part of the whole terminal 100 by various interfaces and lines, executes various functions of the terminal 100 and processes data by running or executing the instructions stored in the memory 120 and calling the data stored in the memory 120, thereby overall controlling the terminal device. Optionally, the processor 170 can include one or more processing units; preferably, the processor 170 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 170. In some embodiments, the processor, the memory, can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips respectively. The processor 170 can also be used to generate corresponding operation control signals to the corresponding components of the computing processing device, read and process the data in the software, especially read and process the data and programs in the memory 120, so that each functional module therein executes corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0124] The memory 120 can be used to store software codes related to the data processing method, and the processor 170 can execute the steps of the data processing method of the chip, or can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.

[0125] The RF unit 110 (optional) can be used to receive and send signals in the process of information or communication, for example, after receiving the downlink information of the base station, the processor 170 processes it; in addition, the uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0126] In the embodiments of the present application, the RF unit 110 can send images to the server 200 and receive the processing results sent by the server 200.

[0127] It should be understood that the RF unit 110 is optional, which can be replaced by other communication interfaces, for example, it can be a network interface.

[0128] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system.

[0129] The terminal 100 also includes an external interface 180, which can be a standard Micro USB interface, or a multi-pin connector, and can be used to connect the terminal 100 and other devices for communication, or can be used to connect a charger to charge the terminal 100.

[0130] Although not shown, the terminal 100 can also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described here. Some or all of the methods described below can be applied in the terminal 100 as shown. Figure 1D

[0131] ​Next, the product form of the server 200 is described. Figure 1C The product form of the server 200 is described.

[0132] Figure 2 A structural diagram of the server 200 is provided, as shown in the figure. Figure 2 The server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate through the bus 201.

[0133] The bus 201 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 In the figure, only one thick line is used, but it does not mean that there is only one bus or one type of bus.

[0134] The processor 202 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0135] The memory 204 can include a volatile memory, such as a random access memory (RAM). The memory 204 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard drive (HDD), or a solid state drive (SSD).

[0136] The memory 204 can be used to store software code related to the data processing method, and the processor 202 can execute the steps of the chip data processing method or schedule other units to realize the corresponding functions.

[0137] It should be understood that the terminal 100 and the server 200 described above can be centralized or distributed devices, and the processors (for example, the processor 170 and the processor 202) in the terminal 100 and the server 200 can be hardware circuits (for example, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processing (DSP), a microprocessor, a microcontroller, or the like) or a combination of the hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, or the like, or a hardware system without an instruction execution function, such as an ASIC, an FPGA, or the like, or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0138] It should be understood that the steps related to the model inference process in the embodiments of the present application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and the server is not limited to the architecture of the processor combined with the memory described above. The following will be described in combination with Figure 3 The system architecture provided by the embodiments of the present application will be described in detail.

[0139] Figure 3 The system architecture provided by the embodiments of the present application will be described in detail. Figure 3 As shown in the figure, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0140] The execution device 510 includes a computing module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The target model / rule 501 can be included in the computing module 511, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0141] The execution device 510 can be a terminal device or a server running an image processing application, as described above.

[0142] The data acquisition device 560 is used to acquire training samples. The training samples can be multiple images, etc. After acquiring the training samples, the data acquisition device 560 stores the training samples in the database 530.

[0143] The training device 520 can obtain the target model / rule 501 by training a neural network (for example, a neural network in the embodiments of the present application, etc.) based on the training samples maintained in the database 530.

[0144] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.

[0145] It should be noted that in actual application, the training samples maintained in the database 530 can not all come from the collection of the data collection device 560, but can also be received from other devices. In addition, it should be noted that the training device 520 can not completely train the target model / rule 501 based on the training samples maintained in the database 530, but can also obtain training samples from the cloud or other places for model training, and the above description should not be regarded as a limitation of the embodiments of the present application.

[0146] The target model / rule 501 trained by the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in the figure, which can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., and can also be a server, etc. Figure 3

[0147] Specifically, the training device 520 can deliver the trained model to the execution device 510.

[0148] In the Figure 3 , the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices, and a user can input data (such as images in the embodiments of the present application, etc.) to the I / O interface 512 through the client device 540.

[0149] The pre-processing module 513 and the pre-processing module 514 are used for pre-processing the input data received by the I / O interface 512. It should be understood that there can be no pre-processing module 513 and pre-processing module 514 or only one pre-processing module. When there is no pre-processing module 513 and pre-processing module 514, the input data can be directly processed by the calculation module 511.

[0150] In the pre-processing of the input data by the execution device 510, or in the calculation and other related processing of the calculation module 511 of the execution device 510, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained by corresponding processing in the data storage system 550.

[0151] ​Finally, the I / O interface 512 provides the processing result to the client device 540, and thus to the user.

[0152] In Figure 3 In the case shown, the user can manually give the input data, which can be operated through the interface provided by the I / O interface 512. In another case, the client device 540 can automatically send the input data to the I / O interface 512, and if the user requires authorization for the client device 540 to automatically send the input data, the user can set the corresponding permission in the client device 540. The user can view the result output by the execution device 510 in the client device 540, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 540 can also act as a data collection end to collect the input data input into the I / O interface 512 and the output result output from the I / O interface 512 as new sample data, and store them into the database 530. Of course, the collection can also be performed without the client device 540, and the I / O interface 512 can directly store the input data input into the I / O interface 512 and the output result output from the I / O interface 512 as new sample data into the database 530.

[0153] It is worth noting that, Figure 3 The above-mentioned system architecture is only a schematic diagram of a system architecture provided by the embodiments of the present application, and the positional relationship between the devices, components, modules, etc. shown in the diagram does not constitute any limitation. For example, in Figure 3 , the data storage system 550 is an external memory relative to the execution device 510, and in other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the above-mentioned execution device 510 can be deployed in the client device 540.

[0154] From the inference side of the model:

[0155] In the embodiments of the present application, the computing module 511 of the execution device 520 can obtain the code stored in the data storage system 550 to implement the steps related to the model inference process in the embodiments of the present application.

[0156] In the embodiments of the present application, the computing module 511 of the execution device 520 can include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with an execution instruction function, such as a CPU, a DSP, etc., or a hardware system without an execution instruction function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an execution instruction function and the hardware system with an execution instruction function.

[0157] Specifically, the computing module 511 of the execution device 520 can be a hardware system with an execution instruction function. The steps related to the model inference process provided in the embodiments of the present application can be software codes stored in a memory. The computing module 511 of the execution device 520 can obtain the software codes from the memory and execute the obtained software codes to implement the steps related to the model inference process provided in the embodiments of the present application.

[0158] It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system without an execution instruction function and a hardware system with an execution instruction function. Part of the steps related to the model inference process provided in the embodiments of the present application can also be implemented by the hardware system without an execution instruction function in the computing module 511 of the execution device 520, which is not limited here.

[0159] From the training side of the model:

[0160] In the embodiments of the present application, the training device 520 can obtain the codes stored in the memory (not shown in the present application) to implement the steps related to the model training in the embodiments of the present application. Figure 3

[0161] ​In the embodiments of the present application, the training device 520 can include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general processor, a digital signal processing (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with an instruction execution function, such as a CPU, a DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0162] It should be understood that the training device 520 can be a combination of a hardware system without an instruction execution function and a hardware system with an instruction execution function, and the part of the steps related to the neutral model training provided in the embodiments of the present application can also be implemented by the hardware system without an instruction execution function in the training device 520, which is not limited here.

[0163] II. Image processing cloud service provided by the server

[0164] In a possible implementation, the server can provide the terminal with an image processing service through an application programming interface (API).

[0165] In this case, the terminal device can send relevant parameters (such as images) to the server through the API provided by the cloud, the server can obtain the processing result based on the received parameters, and return the processing result to the terminal.

[0166] The description of the terminal and the server can be described in the above embodiments, which will not be repeated here.

[0167] For example Figure 4 The process of using an image processing cloud service provided by a cloud platform is shown.

[0168] 1. Open and purchase the content review service.

[0169] 2. The user can download the software development kit (SDK) corresponding to the content review service. Usually, the cloud platform provides multiple development versions of the SDK for the user to select according to the development environment requirements, such as a JAVA version of the SDK, a python version of the SDK, a PHP version of the SDK, an Android version of the SDK, etc.

[0170] 3. The user downloads the corresponding version of the SDK to the local according to the demand, imports the SDK project to the local development environment, and configures and debugs in the local development environment. The local development environment can also develop other functions, so as to form an application that integrates image processing class capabilities.

[0171] 4. In the process of using the image processing class application, when image processing is needed, the API call of image processing can be triggered. When the application triggers the image processing function, an API request is initiated to the running instance of the image processing class service in the cloud environment, wherein the image is carried in the API request. The running instance in the cloud environment processes the image and obtains the processing result.

[0172] 5. The cloud environment returns the processing result to the application, thereby completing a method call provided by the embodiment of the application.

[0173] For example, the data processing method provided by the embodiment of the application can be applied to an ADAS / ADS visual perception system.

[0174] Since the embodiment of the application involves the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the embodiment of the application will be introduced first.

[0175] (1) Neural network

[0176] The neural network can be composed of neural units, and the neural unit can be an operation unit with xs (i.e., input data) and intercept 1 as inputs. The output of the operation unit can be:

[0177]

[0178] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. The neural network is a network formed by connecting a plurality of single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be a region composed of a plurality of neural units.

[0179] (2) Convolutional Neural Network (CNN) is a kind of deep neural network with convolutional structure. CNN contains a feature extractor composed of convolutional layers and subsampling layers, which can be regarded as a filter. The convolutional layer refers to the neuron layer in CNN that performs convolution processing on the input signal. In the convolutional layer of CNN, a neuron can be connected only with part of the adjacent layer neurons. In a convolutional layer, there are usually several feature planes, each of which can be composed of some rectangular arranged neural units. The neural units in the same feature plane share weights, and the shared weights are the convolution kernel. The shared weights can be understood as the way of extracting features regardless of the position. The convolution kernel can be formalized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning in the training process of CNN. In addition, the direct benefit of shared weights is to reduce the connections between the layers of CNN, and at the same time, to reduce the risk of overfitting.

[0180] CNN is a very common neural network, and the structure of CNN will be introduced in detail below. Figure 5A As described in the foregoing basic concept introduction, CNN is a kind of deep neural network with convolutional structure, which is a deep learning architecture. Deep learning architecture refers to learning at multiple levels on different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to the input image.

[0181] As shown in Figure 5A , the convolutional neural network (CNN) 200 can include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230.

[0182] Convolutional layer / pooling layer 220:

[0183] Convolutional layer:

[0184] As shown in Figure 5AThe illustrated convolutional / pooling layers 220 can include layers such as examples 221-226, for example: in one implementation, 221 is a convolutional layer, 222 is a pooling layer, 223 is a convolutional layer, 224 is a pooling layer, 225 is a convolutional layer, and 226 is a pooling layer; in another implementation, 221 and 222 are convolutional layers, 223 is a pooling layer, 224 and 225 are convolutional layers, and 226 is a pooling layer. That is, the output of a convolutional layer can serve as input to a subsequent pooling layer, or as input to another convolutional layer to continue the convolutional operation.

[0185] The internal working principle of a convolutional layer will be described below using convolutional layer 221 as an example.

[0186] Convolutional layer 221 can include a plurality of convolutional operators, also referred to as kernels, which function as filters that extract specific information from an input image matrix in image processing. A convolutional operator can be essentially a weight matrix, which is usually predefined. During the convolutional operation on an image, the weight matrix is usually processed on the input image along the horizontal direction one pixel after another (or two pixels after another, etc., depending on the value of the stride), thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image, and the weight matrix extends to the entire depth of the input image during the convolutional operation. Therefore, convolution with a single weight matrix produces a single-depth-dimension convolutional output, but most cases do not use a single weight matrix, but rather apply a plurality of weight matrices of the same size (rows x columns), i.e., a plurality of homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as being determined by the "plurality" described above. Different weight matrices can be used to extract different features from the image, such as a weight matrix for extracting edge information of the image, another weight matrix for extracting a specific color of the image, and yet another weight matrix for blurring noise in the image that is not needed, etc. The plurality of weight matrices are of the same size (rows x columns), and the feature maps extracted by the plurality of weight matrices of the same size are also of the same size. The plurality of feature maps of the same size are combined to form the output of the convolutional operation.

[0187] The weight values in these weight matrices need to be obtained through a large amount of training in actual applications. The weight matrices formed by the weight values obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0188] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0189] Pooling layer:

[0190] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 5A Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0191] Fully connected layer 230:

[0192] After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize fully connected layers 230 to generate one or a set of outputs representing the required number of classes. Therefore, the fully connected layers 230 can include multiple hidden layers (such as...). Figure 5A As shown in 231, 232 to 23n), the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0193] After the multiple hidden layers in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200, is the output layer 240, which has a loss function similar to the classification cross-entropy, and is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 is completed (as shown in FIG. 2A), the back propagation (as shown in FIG. 2B) starts to update the weight values and biases of the layers mentioned above, so as to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result. Figure 5A The propagation from 210 to 240 is forward propagation, and the back propagation from 240 to 210 starts to update the weight values and biases of the layers mentioned above. Figure 5A The propagation from 210 to 240 is forward propagation, and the back propagation from 240 to 210 starts to update the weight values and biases of the layers mentioned above.

[0194] It should be noted that the convolutional neural network 200 shown in FIG. 2A is only an example of a convolutional neural network, and in specific applications, the convolutional neural network can also exist in the form of other network models, for example, only including a part of the network structure shown in FIG. 2A, for example, the convolutional neural network used in the embodiments of the present application can only include the input layer 210, the convolutional layer / pooling layer 220 and the output layer 240. Figure 5A Figure 5A It should be noted that the convolutional neural network 100 shown in FIG. 1A is only an example of a convolutional neural network, and in specific applications, the convolutional neural network can also exist in the form of other network models, for example, as shown in FIG. 1B, multiple convolutional layers / pooling layers are parallel, and the features extracted by the multiple convolutional layers / pooling layers are all input to the fully connected layer 130 for processing.

[0195] It should be noted that the convolutional neural network 100 shown in FIG. 1A is only an example of a convolutional neural network, and in specific applications, the convolutional neural network can also exist in the form of other network models, for example, as shown in FIG. 1B, multiple convolutional layers / pooling layers are parallel, and the features extracted by the multiple convolutional layers / pooling layers are all input to the fully connected layer 130 for processing. Figure 5A Figure 5B

[0196] (3) Deep Neural Network

[0197] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no specific measurement standard for "many". From the division of DNN according to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected,

[0198] That is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. Although the DNN looks very complex, in terms of the work of each layer, it is actually not complex. In simple terms, it is as follows: wherein, is the input vector, is the output vector, ​​​is the bias vector, W is the weight matrix (also called coefficient), and a() is the activation function. Each layer is simply a linear combination of the input vector After such a simple operation, the output vector is obtained. Since the DNN has many layers, the number of coefficients W and bias vectors is also large. These parameters are defined in the DNN as follows: taking the coefficient W as an example: suppose in a three-layer DNN, the linear coefficient from the fourth neuron of the second layer to the second neuron of the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary: the coefficient from the kth neuron of the L-1th layer to the jth neuron of the Lth layer is defined as It should be noted that the input layer has no W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. In theory, the more parameters a model has, the higher its complexity and "capacity", which means it can complete more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, and the ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by the vectors W of many layers).

[0199] (4) Loss function

[0200] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is truly intended to be predicted, the weight vector of each layer of the neural network can be updated according to the difference between the current network prediction value and the truly intended target value. (Of course, before the first update, there is usually an initialization process, i.e., pre-configuring parameters for each layer in the deep neural network), for example, if the network's prediction value is high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the truly intended target value or a value very close to the truly intended target value. Therefore, it is necessary to define "how to compare the difference between the prediction value and the target value" in advance, which is the loss function or the objective function, which is an important equation for measuring the difference between the prediction value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize this loss.

[0201] (5) Backpropagation algorithm

[0202] The convolutional neural network can adopt a back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model in the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal until the output generates an error loss, and the error loss information is propagated backward to update the parameters in the initial super-resolution model, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain optimal parameters of the super-resolution model, such as a weight matrix.

[0203] (6) Peak signal-to-noise ratio (PSNR): an engineering term representing the ratio of the maximum possible power of a signal to the power of the disruptive noise affecting its representation accuracy. PSNR is often used as a measurement method of signal reconstruction quality in the field of image processing, and is usually simply defined by the mean square error. Generally speaking, the higher the PSNR, the smaller the difference from the true value.

[0204] (7) Structural similarity (SSIM): an index for measuring the similarity between two images. Compared with PSNR, SSIM is more in line with the judgment of human eyes on image quality in measuring image quality.

[0205] (8) Look-up table (LUT): a data structure that replaces an array or associative array of runtime calculations with a simple query operation. Its essence is an indexing technique that trades computation time for storage space through pre-computation.

[0206] (9) Image resampling: a process of resampling pixels in an image according to any geometric transformation and generating a new image. Common geometric transformations include upsampling, downsampling, rotation, tilting, etc.

[0207] (10) Interpolation: the most commonly used resampling method, which usually calculates the weight according to the specified resampling function and performs weighted average on the surrounding pixels. Common interpolation methods include Nearest Neighbor Interpolation, Bilinear Interpolation, Bicubic Interpolation, etc.

[0208] (11) Resampling function: A type of mapping function from distance to weight. Generally, the smaller the distance, the larger the weight. Commonly used resampling functions include nearest neighbor function, linear function, cubic function, etc.

[0209] (12) Local area: In the interpolation method, the range of surrounding pixels for weighted averaging is generally taken as a symmetrical area. For example, in bidirectional linear interpolation, the local area size is 2×2. Then, for a pixel with coordinates (0,0), its local pixel coordinates are (0,0), (0,1), (1,0), (1,1).

[0210] Image resampling is a major research area in low-level computer vision. In recent years, methods based on convolutional neural networks (CNNs) have achieved remarkable results in image resampling performance. Generally, more complex CNN networks tend to achieve better results. Complex structures and relatively large parameters allow for better learning of the characteristics of resampling under different geometric transformations from the data.

[0211] Existing techniques for image resampling can input coordinate offsets into a fixed resampling function to obtain the weight value of the local pixel in the current coordinate system corresponding to each pixel in the output image.

[0212] However, existing interpolation methods all use fixed resampling functions, such as nearest neighbor functions, linear functions, and cubic functions. The calculated weights are only related to the coordinate offset and are independent of the input image. Therefore, these interpolation methods struggle to preserve the unique texture details of different images, resulting in poor image resampling accuracy.

[0213] To address the aforementioned problems, embodiments of this application provide a data processing method. The data processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0214] Reference Figure 6 , Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application, such as... Figure 6 As shown in the embodiment of this application, a data processing method may include steps 601 to 604, which are described in detail below.

[0215] 601. Obtain a first image, wherein the first image includes a plurality of first pixels;

[0216] The first image can be an original image for image resampling, and can include a plurality of first pixel points, which can be part of the pixel points in the first image. For example, after image resampling is performed on the first image, a resampled image can be obtained, and each pixel point in the resampled image can be calculated based on pixel values of a plurality of associated pixel points in the first image. The plurality of first pixel points can be pixel points associated with a pixel point (for example, a target pixel point) in the resampled image.

[0217] 602. According to the first image, a distribution parameter corresponding to each first pixel point is obtained through a target mapping relationship. The distribution parameter is related to a pixel value distribution of the pixel points around the first pixel point. The target mapping relationship is a neural network, or the target mapping relationship includes pixel combination types of the pixel points around the first pixel point and a distribution parameter corresponding to each pixel combination type. The distribution parameter corresponding to each pixel combination type is an output result obtained by processing pixel values corresponding to each pixel combination type based on the neural network. The distribution parameter is used to determine a corresponding resampling function.

[0218] In a possible implementation, according to the first image, a distribution parameter corresponding to each first pixel point is obtained through a target mapping relationship. The distribution parameter is used to determine a corresponding resampling function. The distribution parameter can be used as a parameter in the resampling function, and different distribution parameters can obtain different resampling functions.

[0219] In the prior art, the same resampling function is used for each pixel point. In the embodiment of the present application, the distribution parameter of each pixel point is determined based on the image through the neural network to adjust the resampling function, so that one pixel point can correspond to one resampling function, and the distribution parameter is obtained based on the pixel value distribution around the pixel point. Therefore, the determined resampling function can more accurately quantify the relationship between the pixel bias and the weight, and can learn the edge information of the image, thereby improving the quality of the resampled image.

[0220] In a possible implementation, the target mapping relationship is a neural network.

[0221] In a possible implementation, the target mapping relationship includes pixel combination types of the pixel points around the first pixel point and a distribution parameter corresponding to each pixel combination type. The distribution parameter corresponding to each pixel combination type is an output result obtained by processing pixel values corresponding to each pixel combination type based on the neural network.

[0222] In a possible implementation, the distribution parameter corresponding to the pixel combination type satisfied by the pixel values of the plurality of pixel points at the preset positions near each of the first pixel points in the first image can be obtained through the target mapping relationship.

[0223] The pixel combination type can be related to the position (for example, the pixel points in the square region near the selected pixel point, or the pixel points in the vertical long strip region), the size of the pixel point, and the spatial arrangement relationship of the pixel point. In this case, the pixel values of the plurality of pixel points and the positions (for example, the absolute positions or the relative positions between the pixel points) of the pixel points corresponding to the pixel values in the image are input into the neural network.

[0224] In actual use, for a pixel point, the pixel combination type satisfied by the pixel values of the pixel points near the pixel point can be determined first, and the distribution parameter corresponding to the satisfied pixel combination type can be determined from the target mapping relationship. That is, because the input is a limited pixel value combination, the neural network can not be stored, but only the corresponding relationship between the input and the output of the neural network is stored, thereby reducing the storage cost (without storing the neural network) and the calculation cost (without performing the inference process of the neural network). Compared with the existing interpolation algorithm and the CNN-based method, a better trade-off between performance and power consumption (or performance and runtime delay) is achieved, and better image resampling performance can be achieved with smaller power consumption and runtime delay.

[0225] In a possible implementation, the resampling function represents the corresponding relationship between the coordinate bias and the weight, and the distribution parameter is used to constrain the distribution relationship between the pixel bias and the weight on at least one of the following information:

[0226] The transverse and longitudinal correlation coefficient, the transverse standard deviation, and the longitudinal standard deviation.

[0227] In a possible implementation, the resampling function is as follows:

[0228]

[0229] wherein the d represents the coordinate bias, the p, v X , s Y is the distribution parameter, and the s is the covariance matrix obtained from the distribution parameter, wherein the p is the transverse and longitudinal correlation coefficient, the s X is the transverse standard deviation, and the s Y is the longitudinal standard deviation. The specific visualization results of the p, s X , s Y may refer to the following table. Figure 9

[0230] ​For example, based on the target geometric transformation, the mapping relationship between the target 2D coordinate system and the current 2D coordinate system can be calculated to obtain the offset value of the corresponding coordinate in the current coordinate system for each grid coordinate in the output image. Simultaneously, a local area size is selected to specify how many pixels from the input image are used to calculate the pixel value of each output image. For instance, if the geometric transformation is to enlarge the image by two times, a local area of ​​size 2×2 is used, and the offset values ​​between the target pixel coordinates and the pixel coordinates in this 2×2 local area are (0.5, 0.5), (-0.5, 0.5), (-0.5, -0.5), and (0.5, -0.5), respectively.

[0231] For example, the following anisotropic Gaussian function can be selected:

[0232]

[0233] Where d represents the coordinate offset, and Σ is a coordinate system consisting of (ρ, σ) X ,σ Y The parameterized covariance matrix, such a set of hyperparameter values ​​(ρ, σ) X ,σ Y Different values ​​of the hyperparameter define different resampling functions. By changing the value of the hyperparameter, the shape of the resampling function can be changed, such as... Figure 7 and Figure 8 As shown.

[0234] This application embodiment uses a neural network to predict the values ​​of these hyperparameters. Its input is an image of size H×W, and its output is a tensor of size H×W×M, where M represents the number of hyperparameters. For example, for the anisotropic Gaussian function described above, M is 3. The neural network used can have any structure, provided the input and output dimensions are satisfied.

[0235] 603. Based on the resampling function corresponding to each of the plurality of first pixels, determine the weight corresponding to each first pixel, the weight being used to fuse the pixel values ​​of the plurality of first pixels into the pixel value corresponding to the target pixel; the target pixel is the pixel in the resampled image corresponding to the first image and the pixel corresponding to the plurality of first pixels.

[0236] In one possible implementation, the weight is used to fuse the pixel values ​​of multiple first pixels into the pixel value corresponding to the target pixel. For example, the weight is used to obtain the pixel value corresponding to the target pixel by weighted averaging the pixel values ​​of multiple first pixels.

[0237] For each grid coordinate in the output image, the local pixel values ​​are weighted and averaged to obtain the output pixel value for that coordinate.

[0238] like Figure 10As shown, the embodiment of the present application can obtain the target coordinate pixel by locally weighted average of the pixels. Different from the interpolation method, for each pixel of the input picture, a set of super parameters can be obtained from the resampling function super parameter prediction unit, and each set of super parameters determines a resampling function. Thus, when interpolation is performed, the resampling function used by the embodiment of the present application can be different for each pixel of the input picture, and the local area characteristics of the pixel are adaptively considered. For example Figure 9 As shown, for the flat area, the extension direction of the resampling function is central symmetric, and for the pixel located on the edge of the texture, the corresponding resampling function extends along the extension direction of the edge, and thus better edge preserving performance can be achieved.

[0239] For example, the interpolation unit can be described by the following formula:

[0240] I q =∑ p∈Ω W p I p ,W p =Φ p (d p→q );

[0241] wherein q is the target pixel point coordinate, Ω is the local area taken, p is a pixel coordinate in the area, d p→q represents the coordinate offset from p to q, Φ p is the resampling function of p determined by the prediction unit, and W p is the weight obtained by substituting the coordinate offset into the resampling function. The pixel value I p of the target coordinate point is obtained by weighted average of the pixels I q in Ω.

[0242] 604. Construct a resampling image corresponding to the first image according to the pixel value corresponding to the target pixel point.

[0243] The embodiment of the present application can obtain the prior information related to the local characteristics of the picture by learning from the data, and adaptively change the shape of the resampling function to improve the image processing performance. In contrast, the existing interpolation method uses the same resampling function for all local pixels, ignores the structural characteristics, especially the edge information, and causes the resampling result to be blurred.

[0244] For example, with reference to Figure 11A , Figure 11A is a specific flowchart.

[0245] For example, as shown in Figure 11AAs shown, the first step is to train the hyperparameter prediction network using pairs of low-resolution-high-resolution images, taking the example of upsampling from a low-resolution image of size HxW to a high-resolution image of size 2Hx2W. The neural network used here consists of 6 layers of convolutional layers with kernel size 3x3, which can have any implementation as long as the input is HxW and the output is HxWxM, where M is the number of hyperparameters.

[0246] The second step is to use the trained neural network for prediction inference to predict the hyperparameters used to define the resampling function from a given input image, each pixel in the input image is predicted a set of hyperparameters, in this embodiment, an anisotropic Gaussian function is used, which is defined by three hyperparameters (p, s X , s Y ).

[0247] The third step is to substitute the coordinate bias obtained by any geometric transformation into the adaptive resampling function to obtain the resampling weight, and then perform weighted averaging on the pixels in the local area to obtain the new pixel after resampling. It should be noted that the hyperparameters obtained by the hyperparameter prediction unit of the resampling function are applicable to any geometric transformation, not just upsampling.

[0248] In a possible implementation, a first distribution parameter corresponding to a first pixel combination type satisfied by pixel values of a plurality of pixel points at a first preset position near each of the first pixel points in the first image can be obtained through a target mapping relationship; a second distribution parameter corresponding to a second pixel combination type satisfied by pixel values of a plurality of pixel points at a second preset position near each of the first pixel points in the first image can be obtained through the target mapping relationship; and the first distribution parameter and the second distribution parameter are used to fuse to obtain the distribution parameter.

[0249] As shown in Figure 11B , again, train a neural network with input HxW and output HxWxM, the difference is that the neural network can contain one or more branches, each branch processes the pixels of the input image according to a different index mode, and finally averages the results of each branch. Exemplarily, as shown in Figure 11B , the neural network contains S, C, and X branches, each branch corresponds to an index mode and a corresponding resampling function shape as shown in Figure 11C , because these index modes extend in different edge directions, the embodiments of the present application can be called edge-preserving index modes.

[0250] The second step is to traverse all possible inputs of the neural network, and store different branches as look-up tables (LUTs), such as S branches as S-LUTs. The index of each look-up table is a pixel value, and the stored table value is a hyperparameter value.

[0251] The third step is to use the look-up table to predict the hyperparameter according to the edge-preserving index and edge-preserving aggregation. Figure 11C In the middle, the sampling area of each look-up table is divided into the area with a solid frame and color filling, and the area indicated by the dashed frame. Taking the LUT with the S index mode as an example, the index area indicated by the dashed frame is covered by the edge-preserving aggregation operation, as shown in Figure 11D The edge-preserving aggregation operation can be represented as:

[0252]

[0253] wherein I represents an input image, R j (·) represents j times 180-degree rotation of I, R j -1 (·) represents the inverse transformation of the R j (·) operation.

[0254] In this embodiment, the look-up table acceleration technology is used to accelerate the neural network inference, and the edge-preserving index and edge-preserving aggregation strategy are designed to further extract and perceive the edge information of the local area pixel, so that better power consumption advantage and edge-preserving effect are achieved.

[0255] In addition, from the perspective of the training side, the application also provides an image processing method, which comprises the following steps: acquiring a first image, wherein the first image comprises a plurality of first pixel points; obtaining a distribution parameter corresponding to each first pixel point through a target mapping relationship according to the first image, wherein the distribution parameter is related to the pixel value distribution of the pixel points around the first pixel point; wherein the target mapping relationship is a neural network, or the target mapping relationship comprises a pixel combination type of the surrounding pixel points and a distribution parameter corresponding to each pixel combination type, wherein the distribution parameter corresponding to each pixel combination type is an output result obtained by processing the pixel value corresponding to each pixel combination type based on the neural network; the distribution parameter is used to determine a corresponding resampling function; determining a weight corresponding to each first pixel point according to the resampling function corresponding to each first pixel point in the plurality of first pixel points, wherein the weight is used to fuse the pixel values of the plurality of first pixel points into a pixel value corresponding to a target pixel point; the target pixel point is a pixel point corresponding to the plurality of first pixel points in a resampled image corresponding to the first image; constructing a resampled image corresponding to the first image according to the pixel value corresponding to the target pixel point; and updating the neural network according to the resampled image and a corresponding true value image.

[0256] The feedforward process about the training process can refer to the description of the above embodiments Figure 6 Corresponding description is not repeated here.

[0257] In this embodiment, the loss function selected can be L2 norm loss.

[0258] In a possible implementation, the resampling function represents a corresponding relationship between the coordinate bias and the weight, and the distribution parameter is used to constrain the distribution relationship of the pixel bias and the weight on at least one of the following information:

[0259] The lateral and longitudinal correlation coefficients, the lateral standard deviation, and the longitudinal standard deviation.

[0260] In a possible implementation, the resampling function is as follows:

[0261]

[0262] Wherein, the d represents the coordinate bias, the p, s X , s Y is the distribution parameter, and the s is the covariance matrix obtained by the distribution parameter.

[0263] In a possible implementation, the weight is used to fuse the pixel values of the plurality of first pixel points into the pixel value corresponding to the target pixel point, including: the weight is used to obtain the pixel value corresponding to the target pixel point by weighted average of the pixel values of the plurality of first pixel points.

[0264] Referring to Figure 12 , Figure 12 The structure of a data processing apparatus provided by the embodiments of the present application is shown in Figure 12 The data processing apparatus 1200 provided by the embodiments of the present application includes:

[0265] The acquisition module 1201 is configured to acquire a first image, and the first image includes a plurality of first pixel points.

[0266] The specific description of the acquisition module 1201 can refer to the description of step 601 in the above embodiments, and is not repeated here.

[0267] The processing module 1202 is configured to obtain, according to the first image, a distribution parameter corresponding to each of the first pixel points through a target mapping relationship, the distribution parameter being related to a pixel value distribution of the pixel points around the first pixel point; wherein the target mapping relationship is a neural network, or the target mapping relationship includes pixel combination types of the pixel points around the first pixel point and a distribution parameter corresponding to each of the pixel combination types, wherein the distribution parameter corresponding to each of the pixel combination types is an output result obtained by processing a pixel value corresponding to each of the pixel combination types based on the neural network; and the distribution parameter is used to determine a corresponding resampling function.

[0268] According to the resampling function corresponding to each of the first pixel points, a weight corresponding to each of the first pixel points is determined, and the weight is used to fuse pixel values of the first pixel points into a pixel value corresponding to a target pixel point; and the target pixel point is a pixel point corresponding to the first image in a resampling image of the first image.

[0269] According to the pixel value corresponding to the target pixel point, a resampling image corresponding to the first image is constructed.

[0270] Specific descriptions about the processing module 1202 can be referred to the descriptions of steps 602 to 604 in the above embodiments, which will not be repeated here.

[0271] In a possible implementation, the resampling function represents a corresponding relationship between a coordinate bias and a weight, and the distribution parameter is used to constrain a distribution relationship between the pixel bias and the weight on at least one of the following information:

[0272] a horizontal and vertical correlation coefficient, a horizontal standard deviation, and a vertical standard deviation.

[0273] In a possible implementation, the resampling function is as follows:

[0274]

[0275] wherein d represents a coordinate bias, and p, s X , s Y are distribution parameters, and s is a covariance matrix obtained from the distribution parameters.

[0276] In a possible implementation, the weight is used to fuse pixel values of the first pixel points into a pixel value corresponding to a target pixel point, including:

[0277] The weight is used to obtain the pixel value corresponding to the target pixel point through weighted average of the pixel values of the first pixel points.

[0278] In a possible implementation, the processing module is specifically configured to obtain, according to a pixel combination type satisfied by pixel values of a plurality of pixel points at a preset position near each first pixel point in the first image, a distribution parameter corresponding to the satisfied pixel combination type through a target mapping relationship.

[0279] In a possible implementation, the processing module is specifically configured to obtain, according to a first pixel combination type satisfied by pixel values of a plurality of pixel points at a first preset position near each first pixel point in the first image, a first distribution parameter corresponding to the first pixel combination type through a target mapping relationship.

[0280] obtain, according to a second pixel combination type satisfied by pixel values of a plurality of pixel points at a second preset position near each first pixel point in the first image, a second distribution parameter corresponding to the second pixel combination type through a target mapping relationship; and the first distribution parameter and the second distribution parameter are used to obtain the distribution parameter through fusion.

[0281] In addition, an embodiment of the present application further provides a data processing apparatus, comprising:

[0282] an acquisition module configured to acquire a first image, the first image comprising a plurality of first pixel points;

[0283] a processing module configured to obtain, according to the first image, a distribution parameter corresponding to each first pixel point through a target mapping relationship, the distribution parameter being related to a pixel value distribution of pixel points around the first pixel point; wherein the target mapping relationship is a neural network, or the target mapping relationship comprises pixel combination types of the pixel points around the first pixel point and a distribution parameter corresponding to each pixel combination type, wherein the distribution parameter corresponding to each pixel combination type is an output result obtained by processing a pixel value corresponding to each pixel combination type based on the neural network; and the distribution parameter is used to determine a corresponding resampling function;

[0284] determine a weight corresponding to each first pixel point according to a resampling function corresponding to each first pixel point among the plurality of first pixel points, the weight being used to fuse pixel values of the plurality of first pixel points into a pixel value corresponding to a target pixel point; and the target pixel point is a pixel point corresponding to the plurality of first pixel points in a resampling image corresponding to the first image.

[0285] construct the resampling image corresponding to the first image according to the pixel value corresponding to the target pixel point.

[0286] an updating module configured to update the neural network according to the resampling image and a corresponding true value image.

[0287] In one possible implementation, the resampling function represents the correspondence between coordinate biases and weights, and the distribution parameters are used to constrain the distribution relationship between the pixel biases and the weights on at least one of the following information:

[0288] Horizontal and vertical correlation coefficients, horizontal standard deviation, and vertical standard deviation.

[0289] In one possible implementation, the resampling function is as follows:

[0290]

[0291] Wherein, d represents the coordinate offset, and ρ,v X ,σ Y Let Σ be the distribution parameter, and let Σ be the covariance matrix obtained from the distribution parameter.

[0292] In one possible implementation, the processing module is specifically used to use the weight to obtain the pixel value corresponding to the target pixel by weighted averaging of the pixel values ​​of multiple first pixels.

[0293] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 13 , Figure 13 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1300 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device, or a server, etc., and is not limited thereto. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303, and a memory 1304 (wherein the execution device 1300 may have one or more processors 1303). Figure 13 (Taking a processor as an example), processor 1303 may include application processor 13031 and communication processor 13032. In some embodiments of this application, receiver 1301, transmitter 1302, processor 1303 and memory 1304 may be connected via bus or other means.

[0294] Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303. A portion of memory 1304 may also include non-volatile random access memory (NVRAM). Memory 1304 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0295] The processor 1303 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together by a bus system, which can include a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, various buses are referred to as a bus system in the figure.

[0296] The method disclosed in the embodiments of the present application can be applied to or implemented by the processor 1303. The processor 1303 can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the above method can be completed by hardware integrated logic circuits in the processor 1303 or by instructions in the form of software. The processor 1303 described above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller. It can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1303 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the storage 1304, and the processor 1303 reads the information in the storage 1304 and combines the hardware to complete the steps related to the model inference process in the above method.

[0297] The receiver 1301 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1302 can be used to output digital or character information through the first interface; the transmitter 1302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1302 can also include a display device such as a display screen.

[0298] The embodiments of the present application also provide a training device, please refer to Figure 14 , Figure 14is a structural schematic diagram of a training device provided by an embodiment of the present application. Specifically, the training device 1400 is implemented by one or more servers. The training device 1400 can have a large difference due to different configurations or performances, and can include one or more central processing units (CPUs) 1414 (for example, one or more processors) and a memory 1432, one or more storage media 1430 (for example, one or more mass storage devices) storing an application program 1442 or data 1444. The memory 1432 and the storage media 1430 can be temporary storage or persistent storage. The program stored in the storage media 1430 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the training device. Further, the central processing unit 1414 can be configured to communicate with the storage media 1430 and execute the series of instruction operations in the storage media 1430 on the training device 1400.

[0299] The training device 1400 can further include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458, or one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and the like.

[0300] In an embodiment of the present application, the central processing unit 1414 is configured to perform the actions related to model training in the above embodiments.

[0301] An embodiment of the present application further provides a computer program product, which, when running on a computer, causes the computer to perform the steps performed by the above execution device, or causes the computer to perform the steps performed by the above training device.

[0302] An embodiment of the present application further provides a computer readable storage medium, which stores a program for signal processing, and when running on a computer, causes the computer to perform the steps performed by the above execution device, or causes the computer to perform the steps performed by the above training device.

[0303] The execution device, the training device or the terminal device provided by the embodiments of the present application can be a chip, which includes a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device executes the data processing method described in the above embodiments, or so that the chip in the training device executes the data processing method described in the above embodiments. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0304] Specifically, refer to Figure 15 , Figure 15 A structural schematic diagram of the chip provided by the embodiments of the present application is shown in FIG. 15. The chip can be a neural network processor NPU 1500, which is mounted on a host CPU (Host CPU) as a coprocessor and is assigned tasks by the Host CPU. The core part of the NPU is an operation circuit 1503, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 1504.

[0305] In some implementations, the operation circuit 1503 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1503 is a two-dimensional systolic array. The operation circuit 1503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1503 is a general-purpose matrix processor.

[0306] For example, it is assumed that there are an input matrix A, a weight matrix B and an output matrix C. The operation circuit takes corresponding data of the matrix B from the weight memory 1502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1501 and performs matrix operation with the matrix B, and the partial result or final result of the obtained matrix is saved in an accumulator 1508.

[0307] The unified memory 1506 is used to store input data and output data. The weight data is transferred to the weight memory 1502 through the direct memory access controller (DMAC) 1505. The input data is also transferred to the unified memory 1506 through the DMAC.

[0308] The BIU is the bus interface unit 1510, which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1509.

[0309] The bus interface unit 1510 (BIU) is used for the instruction fetch buffer 1509 to obtain instructions from the external memory, and is also used for the direct memory access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0310] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1506, or to transfer the weight data to the weight memory 1502, or to transfer the input data to the input memory 1501.

[0311] The vector calculation unit 1507 includes a plurality of operation processing units, which further process the output of the operation circuit 1503 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network calculation in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0312] In some implementations, the vector calculation unit 1507 can store the processed output vector to the unified memory 1506. For example, the vector calculation unit 1507 can apply a linear function; or, a nonlinear function to the output of the operation circuit 1503, such as linear interpolation on the feature planes extracted by the convolutional layer, and again, for example, a vector of accumulated values to generate activation values. In some implementations, the vector calculation unit 1507 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in subsequent layers in the neural network.

[0313] The controller 1504 is connected to the instruction fetch buffer 1509, which is used to store instructions used by the controller 1504;

[0314] The unified memory 1506, the input memory 1501, the weight memory 1502, and the instruction memory 1509 are on-chip memories. The external memory is private to the NPU hardware architecture.

[0315] Any processor mentioned in the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling execution of the above programs.

[0316] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0317] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0318] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0319] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a first image, the first image comprising a plurality of first pixels; obtaining, according to the first image, a distribution parameter corresponding to each of the first pixels through a target mapping relationship, the distribution parameter being related to a pixel value distribution of pixels around the first pixel; wherein the target mapping relationship is a neural network, or the target mapping relationship comprises pixel combination categories of the pixels around the first pixel and a distribution parameter corresponding to each of the pixel combination categories, wherein the distribution parameter corresponding to each of the pixel combination categories is an output result obtained by processing a pixel value corresponding to each of the pixel combination categories based on the neural network; and the distribution parameter is used to determine a corresponding resampling function; determining a weight corresponding to each of the first pixels according to the resampling function corresponding to each of the first pixels, the weight being used to fuse pixel values of the plurality of first pixels into a pixel value corresponding to a target pixel; and the target pixel is a pixel corresponding to the plurality of first pixels in a resampling image corresponding to the first image; constructing the resampling image corresponding to the first image according to the pixel value corresponding to the target pixel.

2. The method of claim 1, wherein, The resampling function represents a corresponding relationship between a coordinate bias and a weight, and the distribution parameter is used to constrain a distribution relationship between the pixel bias and the weight on at least one of the following information: a horizontal and vertical correlation coefficient, a horizontal standard deviation, and a vertical standard deviation.

3. The method according to claim 1 or 2, characterized in that, The resampling function is as follows: where d denotes a coordinate bias, p, s X ,σ Y are distribution parameters, and s is a covariance matrix derived from the distribution parameters.

4. The method according to any one of claims 1 to 3, characterized in that, The weight is used to fuse pixel values of the plurality of first pixels into a pixel value corresponding to a target pixel, comprising: The weight is used to fuse pixel values of the plurality of first pixels into a pixel value corresponding to a target pixel through weighted average.

5. The method according to any one of claims 1 to 4, characterized in that, The distribution parameter corresponding to each of the first pixels is obtained according to the first image through a target mapping relationship, comprising: obtaining, according to a pixel combination category satisfied by pixel values of a plurality of pixels at a preset position near each of the first pixels in the first image, a distribution parameter corresponding to the satisfied pixel combination category through the target mapping relationship.

6. The method according to any one of claims 1 to 4, characterized in that, The distribution parameter corresponding to each of the first pixels is obtained according to the first image through a target mapping relationship, comprising: obtaining, according to a first pixel combination category satisfied by pixel values of a plurality of pixels at a first preset position near each of the first pixels in the first image, a first distribution parameter corresponding to the first pixel combination category through the target mapping relationship; obtaining, according to a second pixel combination category satisfied by pixel values of a plurality of pixels at a second preset position near each of the first pixels in the first image, a second distribution parameter corresponding to the second pixel combination category through the target mapping relationship; and the first distribution parameter and the second distribution parameter are used to be fused to obtain the distribution parameter.

7. An image processing method characterized by, The method comprises: obtaining a first image, the first image comprising a plurality of first pixels; According to the first image, a distribution parameter corresponding to each of the first pixel points is obtained through a target mapping relationship, the distribution parameter being related to a pixel value distribution of pixel points around the first pixel point; wherein the target mapping relationship is a neural network, or the target mapping relationship includes pixel combination types of the pixel points around the first pixel point and a distribution parameter corresponding to each of the pixel combination types, wherein the distribution parameter corresponding to each of the pixel combination types is an output result obtained by processing pixel values corresponding to each of the pixel combination types based on the neural network; and the distribution parameter is used to determine a corresponding resampling function; According to the resampling function corresponding to each of the first pixel points, a weight corresponding to each of the first pixel points is determined, the weight being used to fuse pixel values of the first pixel points into a pixel value corresponding to a target pixel point; the target pixel point being a pixel point corresponding to the first pixel points in a resampling image corresponding to the first image; According to the pixel value corresponding to the target pixel point, the resampling image corresponding to the first image is constructed. According to the resampling image and a corresponding true value image, the neural network is updated.

8. The method of claim 7, wherein, The resampling function represents a corresponding relationship between a coordinate bias and a weight, and the distribution parameter is used to constrain a distribution relationship between the pixel bias and the weight on at least one of the following information: A horizontal and vertical correlation coefficient, a horizontal standard deviation, and a vertical standard deviation.

9. The method according to claim 7 or 8, characterized in that, The resampling function is as follows: where d denotes a coordinate bias, p, s X ,σ Y are distribution parameters, and s is a covariance matrix derived from the distribution parameters.

10. The method according to any one of claims 7 to 9, characterized in that, The weight is used to fuse pixel values of the first pixel points into a pixel value corresponding to a target pixel point, including: The weight is used to obtain the pixel value corresponding to the target pixel point through weighted average of the pixel values of the first pixel points.

11. A data processing apparatus, characterized by Including: An acquisition module is configured to acquire a first image, the first image including a plurality of first pixel points; A processing module is configured to obtain, according to the first image, a distribution parameter corresponding to each of the first pixel points through a target mapping relationship, the distribution parameter being related to a pixel value distribution of pixel points around the first pixel point; wherein the target mapping relationship is a neural network, or the target mapping relationship includes pixel combination types of the pixel points around the first pixel point and a distribution parameter corresponding to each of the pixel combination types, wherein the distribution parameter corresponding to each of the pixel combination types is an output result obtained by processing pixel values corresponding to each of the pixel combination types based on the neural network; and the distribution parameter is used to determine a corresponding resampling function; According to the resampling function corresponding to each of the first pixel points, a weight corresponding to each of the first pixel points is determined, the weight being used to fuse pixel values of the first pixel points into a pixel value corresponding to a target pixel point; the target pixel point being a pixel point corresponding to the first pixel points in a resampling image corresponding to the first image; According to the pixel value corresponding to the target pixel point, the resampling image corresponding to the first image is constructed.

12. The apparatus of claim 11, wherein, The resampling function represents a corresponding relationship between a coordinate bias and a weight, and the distribution parameter is used to constrain a distribution relationship between the pixel bias and the weight on at least one of the following information: Horizontal and vertical correlation coefficients, horizontal standard deviation, and vertical standard deviation.

13. The apparatus of claim 11 or 12, wherein, The resampling function is given by the following formula: where d represents a coordinate bias, p, v X ,σ Y are distribution parameters, and Σ is a covariance matrix derived from the distribution parameters.

14. The apparatus of any one of claims 11 to 13, wherein, The weight is used to fuse the pixel values ​​of multiple first pixels into the pixel value corresponding to the target pixel, including: The weight is used to obtain the pixel value corresponding to the target pixel by weighted averaging of the pixel values ​​of multiple first pixels.

15. The apparatus of any one of claims 11 to 14, wherein, The processing module is specifically used to obtain the distribution parameters corresponding to the satisfied pixel combination types by means of a target mapping relationship, based on the pixel value of multiple pixels located at preset positions near each first pixel in the first image satisfying the pixel combination types.

16. The apparatus of any one of claims 11 to 14, wherein, The processing module is specifically used to obtain the first distribution parameter corresponding to the first pixel combination type based on the first pixel combination type satisfied by the pixel values ​​of multiple pixels located at a first preset position near each first pixel in the first image, through a target mapping relationship. Based on the second pixel combination type satisfied by the pixel values ​​of multiple pixels located at a second preset position near each first pixel in the first image, a second distribution parameter corresponding to the second pixel combination type is obtained through a target mapping relationship; the first distribution parameter and the second distribution parameter are used to fuse to obtain the distribution parameter.

17. A data processing apparatus, characterized by: include: The acquisition module is used to acquire a first image, wherein the first image includes a plurality of first pixels; The processing module is configured to obtain distribution parameters corresponding to each first pixel point based on the first image and through a target mapping relationship. The distribution parameters are related to the pixel value distribution of pixels surrounding the first pixel point. The target mapping relationship is either a neural network or includes pixel combination types of the surrounding pixels and distribution parameters corresponding to each pixel combination type. The distribution parameters corresponding to each pixel combination type are the output results obtained by processing the pixel values ​​corresponding to each pixel combination type based on a neural network. The distribution parameters are used to determine a corresponding resampling function. Based on the resampling function corresponding to each of the plurality of first pixels, a weight corresponding to each first pixel is determined. The weight is used to fuse the pixel values ​​of the plurality of first pixels into the pixel value corresponding to the target pixel. The target pixel is the pixel in the resampled image corresponding to the first image and the pixel corresponding to the plurality of first pixels. Based on the pixel values ​​corresponding to the target pixels, a resampled image corresponding to the first image is constructed; An update module is used to update the neural network based on the resampled image and the corresponding ground truth image.

18. The apparatus of claim 17, wherein, The resampling function represents the correspondence between coordinate offsets and weights, and the distribution parameters are used to constrain the distribution relationship between the pixel offsets and the weights based on at least one of the following information: Horizontal and vertical correlation coefficients, horizontal standard deviation, and vertical standard deviation.

19. The apparatus of claim 17 or 18, wherein, The resampling function is given by the following formula: where d denotes a coordinate bias, p, s X ,σ Y are distribution parameters, and s is a covariance matrix derived from the distribution parameters.

20. The apparatus of any one of claims 17 to 19, wherein, The processing module is specifically used to use the weight to obtain the pixel value corresponding to the target pixel by weighted averaging of the pixel values ​​of multiple first pixel points.

21. A computer storage medium, comprising, The computer storage medium stores one or more instructions which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1-10.

22. A computer program product, characterised in that, Computer readable instructions are included which, when run on a computer device, cause the computer device to perform the method of any one of claims 1-10.

23. A system comprising at least one processor, at least one memory; the processor, the memory are connected through a communication bus and complete the communication between each other; The at least one memory is configured to store code; The at least one processor is configured to execute the code to perform the method of any one of claims 1-10.

Images