Image processing method, device, system and intelligent device

Abstract

The application provides an image processing method, device and system, which are suitable for the fields of intelligent vehicles, intelligent terminals, smart homes and the like. The method comprises the following steps: acquiring first image data output by an ISP, wherein the first image data is obtained by the ISP from original image data according to target ISP parameter groups, and the first image data is used for a first service; obtaining second image data according to the first image data, wherein the data amount of the second image data is greater than or equal to the data amount of the first image data, and is less than or equal to the data amount of the original image data; and obtaining third image data according to the second image data, wherein the third image data is used for a second service. According to the application, the first image data output by the ISP and used for the first service is processed to obtain the third image data which can be used for the second service, so that the number of services supported by the ISP is increased.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image processing method, apparatus, system, and intelligent device. Background Technology

[0002] An image signal processor (ISP) primarily processes the raw image data acquired by the front-end image sensor and then transmits the processed image data to human vision (HV) or computer vision (CV) related services. An ISP typically contains multiple data channels, each with one ISP parameter group, which needs to be configured according to the specific application requirements.

[0003] Different services typically have different requirements for image quality metrics. An ISP parameter set configured for one service may not meet the needs of other services. For example, an ISP parameter set optimized based on subjective human evaluation, suitable for HV-related services, may not be suitable for CV-related services. Traditionally, different ISP parameter sets are configured on different data channels of the ISP, allowing image data output by the ISP to be used for different services. However, since one data channel can only correspond to one ISP parameter set, and the number of data channels is limited by ISP bandwidth, the services supported by the ISP are limited by the number of data channels.

[0004] Therefore, how to increase the number of services supported by ISPs is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides an image processing method, apparatus, system, and smart device that can increase the number of services supported by an ISP.

[0006] In a first aspect, an image processing method is provided, comprising: acquiring first image data output by an ISP, the first image data being obtained by the ISP processing original image data according to a target ISP parameter set, the first image data being used for a first service, the original image data being acquired by an image sensor; obtaining second image data based on the first image data, the data volume of the second image data being greater than or equal to the data volume of the first image data and less than or equal to the data volume of the original image data; and obtaining third image data based on the second image data, the third image data being used for a second service.

[0007] In this application, on the one hand, by processing the first image data output by the ISP for the first service, a third image data that can be used for the second service is obtained, thereby expanding the services corresponding to the target ISP parameter group in the ISP and increasing the number of services supported by the ISP; on the other hand, in the process of processing the first image data to obtain the third image data, the second image data can be obtained first from the first image data, and then the third image data can be obtained from the second image data. The amount of data in the second image data can be greater than or equal to the amount of data in the first image data and less than or equal to the amount of data in the original image data, which means that the second image data can contain more image data information than the first image data, thereby improving the quality of the third image data.

[0008] Obtaining second image data from first image data can also be described as: obtaining second image data by restoring first image data. Therefore, the second image data can be the first image data, image data from the restoration process, or the restored original image data.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the second image data based on the first image data includes: restoring the first image data based on a first model to obtain the second image data, wherein the first model is obtained based on a second model, and the second model is trained based on image data samples from the first service.

[0010] The image data sample for the first service can be an image data sample related to the first service. Specifically, the image data sample for the first service may include a raw image data sample acquired from the image sensor and a first image data sample obtained by the ISP processing the raw image data sample according to the target ISP parameter set. The first image data sample can be applied to the first service.

[0011] Specifically, a second model is trained using raw image data samples acquired by an image sensor and a first image data sample obtained by an ISP processing the raw image data samples according to a target ISP parameter set. During training, the raw image data samples are used as input data for the training model, and the model parameters are continuously adjusted. The second model is obtained when the difference between the image data output by the training model and the first image data sample meets a preset value.

[0012] The second model is an invertible model, and the first model is obtained from the inverse transformation model of the second model.

[0013] It should be understood that obtaining the second image data from the first image data, as mentioned above, can be interpreted as a process of restoring the first image data. However, due to the high complexity of the ISP algorithm, it is difficult to directly perform inverse operations based on the ISP model to restore the first image data and obtain the second image data. Therefore, in this application, a second model different from the ISP is first trained using image data samples from the first service, and then the first model is obtained through the second model to realize the restoration of the first image data, thereby reducing the complexity of restoring the first image data.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the second model is used to simulate the output of the ISP based on the image data acquired by the image sensor.

[0015] In other words, the second model can obtain the same output when given the same input data as the ISP. It should be understood that, since the second model is not identical to the ISP itself, this application allows for slight differences between the image data simulated by the second model and the image data output by the ISP.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the third image data from the second image data includes: processing the second image data based on a third model to obtain the third image data, wherein the third model is trained according to the requirements of the second service.

[0017] In this application, the second image data can be processed based on the third model to obtain the third image data, and the third model is trained according to the requirements of the second service, so that the quality of the third image data processed by the third model can meet the requirements of the second service.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the first model includes n serially connected computing units and n+1 output terminals. The output of each of the n serially connected computing units serves as the input of the next computing unit. The input of the first computing unit among the n computing units is the first image data. One of the n+1 output terminals outputs the second image data, where n is a positive integer.

[0019] The computational unit can be a radial coupling layer.

[0020] One of the n+1 output terminals is set at the input of the first of the n calculation units, and the other n output terminals are set at the output of each of the n calculation units.

[0021] The first model involved in this application may include n serially connected computing units and n+1 output terminals, and one of the n+1 output terminals outputs second image data, so that during the restoration process of the first image data, the output terminal that outputs the second image data can be selectively determined according to actual needs.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the first model determines the output terminal for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one with the n+1 output terminals.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output terminal that outputs the second image data is 1.

[0024] In this case, a value of 0 for the selection parameter means no output, and a value of 1 means output.

[0025] In a second aspect, an image processing apparatus is provided, comprising: an acquisition module for acquiring first image data output by an ISP, the first image data being obtained by the ISP processing raw image data according to a target ISP parameter set, the first image data being used for a first service, the raw image data being acquired by an image sensor; a first processing module for obtaining second image data based on the first image data, the data volume of the second image data being greater than or equal to the data volume of the first image data and less than or equal to the data volume of the raw image data; and a second processing module for obtaining third image data based on the second image data, the third image data being used for a second service.

[0026] In conjunction with the second aspect, in some implementations of the second aspect, the first processing module is used to restore the first image data based on the first model to obtain the second image data, wherein the first model is obtained based on the second model, and the second model is trained based on the image data samples of the first service.

[0027] In conjunction with the second aspect, in some implementations of the second aspect, the second model is used to simulate the output of the ISP based on the image data acquired by the image sensor.

[0028] In conjunction with the second aspect, in some implementations of the second aspect, the second processing module is used to process the second image data based on a third model to obtain the third image data, wherein the third model is trained according to the requirements of the second service.

[0029] In conjunction with the second aspect, in some implementations of the second aspect, the first model includes n serially connected computing units and n+1 output terminals. The output of each of the n serially connected computing units serves as the input of the next computing unit. The input of the first computing unit among the n computing units is the first image data. One of the n+1 output terminals outputs the second image data, where n is a positive integer.

[0030] In conjunction with the second aspect, in some implementations of the second aspect, the first model determines the output terminal for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one with the n+1 output terminals.

[0031] In conjunction with the second aspect, in some implementations of the second aspect, the value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output terminal that outputs the second image data is 1.

[0032] Thirdly, an image processing apparatus is provided, including an input / output interface, a processor, and a memory. The processor controls the input / output interface to send and receive signals or information, the memory stores a computer program, and the processor calls and runs the computer program from the memory, causing the image processing apparatus to perform an image processing method as described in the first aspect or any possible implementation thereof.

[0033] Fourthly, an image processing system is provided, comprising: an image sensor for acquiring raw image data; an ISP for processing the raw image data according to a target ISP parameter set to obtain first image data, the first image data being used for a first service; and N image processing devices for: obtaining N different second image data based on the first image data, wherein the data size of each of the N different second image data is greater than or equal to the data size of the first image data and less than or equal to the data size of the raw image data; and obtaining N different third image data based on the N different second image data, the N different third image data being used for N different second services, where N is a positive integer greater than or equal to 1.

[0034] Among them, N image processing devices correspond one-to-one with N different second image data, N different second image data correspond one-to-one with N different third image data, and N different third image data correspond one-to-one with N different second services.

[0035] In this application, the first image data can be processed in parallel by N image processing devices, and N different third image data can be output in parallel for N different second services, thereby increasing the number of services supported by the ISP.

[0036] Fifthly, a computer-readable medium is provided that stores program code, which, when run on a computer, causes the computer to perform an image processing method as described in the first aspect or any possible implementation thereof.

[0037] In a sixth aspect, a computer program product is provided, comprising instructions that, when executed on a computer, perform the image processing method as described in the first aspect or any possible implementation thereof.

[0038] In a seventh aspect, a computing device is provided, comprising: at least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory to perform an image processing method as described in the first aspect or any possible implementation thereof.

[0039] Eighthly, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute an image processing method as described in the first aspect or any possible implementation thereof.

[0040] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform an image processing method as described in the first aspect or any possible implementation thereof.

[0041] A ninth aspect provides an electronic device including various modules for performing an image processing method as described in the first aspect or any possible implementation thereof.

[0042] In a tenth aspect, an intelligent driving device is provided, including an image processing apparatus as described in the second aspect or any possible implementation thereof. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of a traditional system architecture for image processing.

[0044] Figure 2 This is a schematic diagram of the ISP data channel provided in an embodiment of this application.

[0045] Figure 3 This is a schematic diagram of the configuration scheme of the ISP data channel provided in the embodiments of this application.

[0046] Figure 4 This is a schematic diagram of the image processing method provided in the embodiments of this application.

[0047] Figure 5 This is a schematic diagram of the structure of the second model provided in the embodiments of this application.

[0048] Figure 6 This is a schematic diagram of the training process of the second model provided in the embodiments of this application.

[0049] Figure 7 This is a schematic diagram of the structure of the first model provided in the embodiments of this application.

[0050] Figure 8 This is a schematic diagram of the structure of the first model provided in the embodiments of this application.

[0051] Figure 9 This is a schematic diagram of the structure of the first model provided in the embodiments of this application.

[0052] Figure 10 This is a schematic diagram of the structure of the third model provided in the embodiments of this application.

[0053] Figure 11 This is a schematic diagram of the structure of a system provided in an embodiment of this application.

[0054] Figure 12 This is a schematic diagram of another system provided in an embodiment of this application.

[0055] Figure 13 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application.

[0056] Figure 14 This is an exemplary block diagram of the hardware structure of the image processing apparatus provided in the embodiments of this application. Detailed Implementation

[0057] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0058] This solution can be applied to fields such as intelligent vehicles, intelligent terminals, and smart homes. For example, it can be applied to scenarios where image processing can be performed using an ISP, such as smartphones, desktop computers, laptops, tablets, wearable devices, home robots, or intelligent driving devices.

[0059] Intelligent driving devices can include road vehicles, water vehicles, air vehicles, industrial equipment, agricultural equipment, or entertainment equipment. For example, an intelligent driving device can be a vehicle, which is a vehicle in a broad sense, including transportation vehicles (such as commercial vehicles, passenger cars, motorcycles, flying cars, trains, etc.), industrial vehicles (such as forklifts, trailers, tractors, etc.), engineering vehicles (such as excavators, bulldozers, cranes, etc.), agricultural equipment (such as lawnmowers, harvesters, etc.), amusement equipment, toy vehicles, etc. The embodiments of this application do not specifically limit the type of vehicle. As another example, an intelligent driving device can be a vehicle such as an airplane or a ship.

[0060] To facilitate understanding, let's first combine... Figures 1 to 3 A brief introduction to the background technology involved in the embodiments of this application is given below.

[0061] Figure 1 This is a schematic diagram of a traditional system architecture for image processing. For example... Figure 1 As shown, the system architecture 100 includes an image sensor 110, an ISP 120, and a service processing module 130.

[0062] The image sensor 110 is used to sense the external environment, acquire raw image data, and then transmit the raw image data to the ISP 120 through a serializer and a deserializer. The image sensor 110 can be, for example, a camera. The ISP 120 processes the raw image data acquired by the image sensor 110, such as performing black level compensation, bad pixel correction, color interpolation, Bayer noise removal, white balance correction, color restoration correction, gamma correction, color space noise removal and edge enhancement, color and contrast enhancement, etc., and then transmits the processed image data to the service processing module 130 for use.

[0063] The business processing module 130 is used to process business. There can be one or more business processing modules 130. The business can be HV-related business or CV-related business. Taking the application in the vehicle field as an example, CV-related business can include target detection, lane line detection, traffic light recognition, semantic segmentation, etc., while HV-related business can include electronic rearview mirror data processing, surround view stitching, and rearview camera data processing, etc.

[0064] It should be understood that transmitting the image data processed by the ISP 120 to the service processing module 130 means transmitting the processed image data to the service processing module 130 corresponding to the service. For example, if the service is lane detection, then the image data processed by the ISP 120 is transmitted to the service processing module 130a related to lane detection. If the service is electronic rearview mirror data processing, then the image data processed by the ISP 120 is transmitted to the service processing module 130b related to electronic rearview mirrors.

[0065] An ISP typically contains multiple data channels, and each data channel can be configured with one ISP parameter group. Figure 2 This is a schematic diagram of the ISP data channel provided in an embodiment of this application, as shown below. Figure 2 As shown, this ISP contains L data channels, and each L data channel can be configured with L sets of ISP parameter groups. It should be understood that each ISP parameter group in the ISP can output one image data point, and each ISP parameter group needs to be configured according to the actual application service. Different services typically have different requirements for image quality indicators, and an ISP parameter group configured for one service may not meet the needs of other services. For example, an ISP parameter group optimized based on subjective human evaluation that is suitable for HV-related services may not meet the needs of CV-related services.

[0066] In traditional solutions, different ISP parameter groups are typically configured on different data channels of the ISP, allowing the image data output by the ISP to be used for different services based on the different ISP parameter groups. For example... Figure 3 As shown, ISP parameter groups can be configured for some data channels to suit CV-related services, and for others to suit HV-related services, so that the image data output by the ISP can be used for different services. However, in the current situation, since one data channel corresponds to only one ISP parameter group, and the number of data channels is limited by the ISP bandwidth, the services supported by the ISP are limited by the number of data channels.

[0067] Furthermore, typically an ISP's data channel can only connect to one image sensor, and a particular service may require multiple image sensors simultaneously to acquire image data from different perspectives. These multiple image sensors correspond to multiple data channels of the ISP. Therefore, adding other services requires occupying the data channels and image sensors corresponding to the current service, reducing the number of image sensors available for that service and thus affecting the acquisition of the image information required by that service.

[0068] Based on this, the present application provides an image processing method that processes image data output by an ISP that is suitable for one service to obtain image data suitable for another service, thereby expanding the services corresponding to a group of ISP parameters in the ISP, thereby increasing the number of services supported by the ISP, and ensuring that the number of image sensors corresponding to each service is not affected.

[0069] Figure 4 This is a schematic diagram of the image processing method provided in an embodiment of this application. For example... Figure 4 As shown, the method 400 includes steps S410 to S430, which will be described in detail below.

[0070] S410, acquire the first image data output by the ISP, which is used for the first service.

[0071] The first image data is obtained by the ISP processing the raw image data according to the target ISP parameter set, which is acquired by the image sensor.

[0072] The aforementioned target ISP parameter group refers to an ISP parameter group on a data channel within an ISP. It should be understood that this application primarily uses the target ISP parameter group as an example; in practice, this application also applies to ISP parameter groups on one or more other data channels within an ISP.

[0073] S420, obtain the second image data based on the first image data.

[0074] The amount of data in the second image data is greater than or equal to the amount of data in the first image data, and less than or equal to the amount of data in the original image data.

[0075] It should be understood that obtaining the second image data from the first image data can also be understood as: obtaining the second image data by restoring the first image data. Based on this, the aforementioned second image data can be the first image data, the restored original image data, or image data between the first image data and the restored original image data. It should be understood that if the second image data is image data between the first image data and the restored original image data, it has a larger data volume than the first image data and can contain more image data information; compared to the original image data, it can reduce the computational load, memory usage, and bandwidth usage in the process of obtaining the second image data from the first image data.

[0076] Optionally, the first image data can be restored based on the first model to obtain the second image data. The first model may include n cascaded computing units and n+1 output terminals. The output of each of the n cascaded computing units serves as the input of the next computing unit. The input of the first computing unit among the n computing units is the first image data, and one of the n+1 output terminals outputs the second image data, where n is a positive integer.

[0077] The computational unit can be a radial coupling layer.

[0078] One of the aforementioned n+1 output terminals is located at the input of the first of the n computing units, and the other n output terminals are located at the output of each of the n computing units. Optionally, the output terminal used to output the second image data among the n+1 output terminals can be determined according to the requirements of the second service, as detailed below.

[0079] The first model can determine the output terminals for the second image data using n+1 selection parameters, with each of the n+1 selection parameters corresponding to one of the n+1 output terminals. In other words, the n+1 selection parameters can be used to control which of the n+1 output terminals does not output, and which outputs only one.

[0080] For example, each of the n+1 selection parameters can have a value of 0 or 1, and the sum of these n+1 selection parameters is 1. The selection parameter corresponding to the output terminal that outputs the second image data has a value of 1. Here, a value of 0 represents no output, and a value of 1 represents output.

[0081] As one implementation, the first model can be derived from a second model, which is used to simulate the output of the ISP based on image data acquired by the image sensor. In other words, the second model can obtain the same output when the input data is the same as that of the ISP. It should be understood that since the second model is not the same as the ISP itself, in this embodiment of the application, the image data simulated by the second model may have slight differences from the image data output by the ISP.

[0082] The second model may include n cascaded computational units, the output of each of the n cascaded computational units serving as the input of the next computational unit, the input of the first computational unit of the n computational units serving as the input of the second model, and the output of the nth computational unit serving as the output of the second model.

[0083] The second model is trained based on image data samples from the first service. The image data samples from the first service can be image data samples related to the first service. Specifically, the image data samples from the first service can include raw image data samples acquired from the image sensor and first image data samples obtained by the ISP processing the raw image data samples according to the target ISP parameter set. These first image data samples can be applied to the first service.

[0084] Specifically, a second model is trained using raw image data samples acquired by an image sensor and a first image data sample obtained by an ISP processing the raw image data samples according to a target ISP parameter set. During training, the raw image data samples are used as input data for training the second model, and the model parameters of the second model are continuously adjusted. The second model is obtained when the difference between the image data output by the second model and the first image data sample meets a preset value.

[0085] The second model is an invertible model. The first model can be derived from the second model or described as follows: the n cascaded computational units of the first model are obtained by inverse transformation of the second model.

[0086] In this application, obtaining the second image data from the first image data can be understood as a process of restoring the first image data. However, since the ISP algorithm is relatively complex, it is difficult to restore the first image data to obtain the second image data by directly performing inverse operations based on the ISP model. Therefore, in this embodiment, a second model different from the ISP is first trained using image data samples from the first service, and then the first model is obtained through the second model to realize the restoration of the first image data, thereby reducing the complexity of restoring the first image data.

[0087] For details on the construction of the second model and the process of obtaining the first model based on the second model, please refer to the following text.

[0088] As another implementation method, the first model can also be trained directly from the image data samples of the first service. That is to say, the first model can be trained directly from the image data samples of the first service. Specifically, during the training process, the first image data samples are used as the input data for training the first model, and the model parameters of the first model are continuously adjusted. The first model is obtained when the difference between the image data output by the first model and the original image data samples meets a preset value.

[0089] S430, obtain third image data based on the second image data, and use the third image data for the second service.

[0090] Optionally, the second image data can be processed based on a third model to obtain third image data, wherein the third model is trained according to the requirements of the second service.

[0091] The requirements for the second service may include performance requirements and resource requirements. Performance requirements refer to the performance metrics required of the processing modules involved in the second service. For example, if the second service is a computer vision (CV) related service, the performance requirements may refer to the accuracy, precision, and recall of the processing modules involved in the CV-related service (e.g., deep neural networks (DNNs) and other perceptual networks) when using third image data as input. If the second service is a human image quality (HV) related service, the performance requirements may refer to the subjective evaluation metrics required for the quality of the output image data when using third image data as input. Resource requirements may include memory bandwidth and data processing latency requirements, which refer to the memory bandwidth and data processing latency requirements used in the process of obtaining the third image data applied to the second service according to the scheme of this application.

[0092] In this embodiment of the application, the second image data can be processed based on the third model to obtain the third image data, and the third model is trained according to the requirements of the second service, so that the quality of the third image data processed by the third model can meet the requirements of the second service.

[0093] Optionally, the output terminal for outputting the second image data and the training of the third model can be determined based on the performance indicators of the second service; or, the output terminal for outputting the second image data and the training of the third model can be determined based on the bandwidth, memory and / or latency indicators of the first model processing the first image data, and the performance indicators of the second service processing.

[0094] The third model may include p cascaded computation units, where the output of each of the p cascaded computation units serves as the input of the next computation unit. The input of the first computation unit in the p cascaded computation units is the second image data, and the output of the p-th computation unit is the third image data, where p is a positive integer.

[0095] It should be understood that the target parameter set in an ISP typically includes a large number of parameters. In practice, only a portion of the parameters in the target parameter set can be used to obtain the first image data, while the functions required by the other portion of the parameters can be achieved through processing by a third model. This means that during the training of the third model, it is also necessary to train the model based on the functions required by the other portion of the parameters, so that these functions can be implemented through the third model. For example, during image processing in the ISP, white balance correction can be omitted initially and implemented during the subsequent processing by the third model.

[0096] The business covered by this application may include HV-related business and CV-related business, etc. For a description of the HV-related business and CV-related business, please refer to the above text.

[0097] Based on this, the first service mentioned above can be one of the HV-related services, and the second service can be one of the CV-related services; or the first service can be one of the CV-related services, and the second service can be one of the HV-related services; or the first service can be the first service among HV-related services, and the second service can be the second service among HV-related services; or the first service can be the first service among CV-related services, and the second service can be the second service among CV-related services.

[0098] In this embodiment, on the one hand, by processing the first image data output by the ISP for the first service, a third image data that can be used for the second service is obtained, thereby expanding the services corresponding to the target ISP parameter group in the ISP and increasing the number of services supported by the ISP. On the other hand, in the process of processing the first image data to obtain the third image data, the second image data can be obtained first from the first image data, and then the third image data can be obtained from the second image data. The amount of data in the second image data can be greater than or equal to the amount of data in the first image data and less than or equal to the amount of data in the original image data. This means that the second image data can contain more image data information than the first image data, thereby improving the quality of the third image data.

[0099] The following example uses the computational unit as an affine coupling layer, combined with... Figures 5 to 10 The model construction of the first model, the second model, and the third model are described in detail.

[0100] Figure 5 This is a schematic diagram of the structure of the second model provided in an embodiment of this application. This second model can simulate the output of an ISP based on image data acquired by an image sensor. In this embodiment, the construction of the second model is primarily for the purpose of obtaining the first model.

[0101] like Figure 5 As shown, the second model includes n cascaded affine coupling layers {F1, F2, ..., F...} k ,...,F n The output of each of the n cascaded affine coupling layers serves as the input to the next affine coupling layer. The input of the first affine coupling layer F1 is the input to the second model, and the input of the nth affine coupling layer F... n The output is the output of the second model. The affine coupling layer possesses reversible properties.

[0102] For the specific structure of each radiation coupling layer, please refer to [link / reference]. Figure 5 The affine coupling layer F1 shown can be represented by the following formula (1):

[0103]

[0104] Where, x a and x b For the input data of the affine coupling layer, taking affine coupling layer F1 as an example, x a and x b It is two parts of image data segmented from the first image data; m a and m b This represents the output data of the affine coupling layer; the symbol ⊙ indicates element-wise multiplication, s(x a ) and t(x a ) is a mapping function. It should be understood that s(x) a ) and t(x a ) can be a linear or nonlinear function, without limitation. For example, s(x) a The expression of ) can be shown in formula (2):

[0105] s(x a )=αx a +β (2)

[0106] Here, α and β are the parameters that can be trained in the second model.

[0107] It should be understood that the above formula (2) is only used as s(x) a This is one example; in practice, other forms can also be used, and there are no restrictions. Additionally, t(x) a The representation of ) and s(x) a )similar.

[0108] Based on the second model structure created above, the model parameters of the second model are trained using image data samples from the first service. The image data samples from the first service may include raw image data samples acquired from an image sensor and first image data samples obtained by processing the raw image data samples by an ISP. These first image data samples can be applied to the first service.

[0109] The following is combined with Figure 6 The training process of the second model is described exemplarily. For example... Figure 6 As shown, the original image data samples can be simultaneously input into the ISP and the second model. Then, the first image data sample output by the ISP and the image data output by the second model are compared, and the difference between the two output data is calculated. The model parameters of the second model are continuously adjusted according to the difference until the difference between the output of the ISP and the output of the second model meets the preset value, thus completing the training of the second model.

[0110] Alternatively, a loss function can be defined in advance, and a second model can be trained based on this loss function. This application embodiment does not limit the specific form of the loss function. For example, the loss function can be as shown in formula (3):

[0111] Loss = ||IMG1 - IMG2||2 (3)

[0112] Wherein, IMG1 is the output of ISP, and IMG2 is the output of the second model.

[0113] It should be understood that the higher the value of the loss function, the greater the difference between the output of the ISP and the output of the second model. During the training process, the value of the loss function can be calculated based on the output of the ISP and the output of the second model, and the model parameters of the second model can be continuously adjusted based on the value of the loss function until the value of the loss function meets the preset value, that is, the training of the second model is completed.

[0114] After determining the second model, the inverse transformation of the second model can be performed based on the reversible property of the affine coupling layer to obtain the first model. Then, the inverse reconstruction of the first image data output by the ISP can be achieved based on the first model. The following section combines... Figures 7 to 9 The first model will be introduced.

[0115] Figure 7 This is a schematic diagram of the structure of the first model provided in an embodiment of this application. The first model is mainly used to reverse-engineer the first image data output by the ISP to obtain more image data information.

[0116] like Figure 7 As shown, the first model includes n cascaded affine coupling layers. With n+1 output terminals, the output of each of the n cascaded affine coupling layers serves as the input of the next affine coupling layer. The first affine coupling layer F of the n affine coupling layers... n -1 The input is the first image data, and one of the n+1 output terminals outputs the second image data, where n is a positive integer.

[0117] Among them, n cascaded affine coupling layers This is achieved by using n cascaded affine coupling layers {F1, F2, ..., F...} k ,...,F n It is obtained by inverting the order of layers.

[0118] Using the nth affine coupling layer F1 of the first model -1 For example, this affine coupling layer is obtained by inverting the first affine coupling layer F1 in the second model. Based on this, the affine coupling layer F1... -1 The specific structure can be represented by the following formula (4):

[0119]

[0120] For example, see Figure 7 It can use n+1 selection parameters {S n ,S n-1 ,...,S k The expression `S0, ...,S1,S0` determines which of the `n+1` output terminals outputs the second image data. Here, `S` is either 0 or 1, where 0 represents no output and 1 represents output, and `S0 + S1 + ... + S` is a constant. n =1 means that one of the n+1 selection parameters is 1, and the others are 0, so as to ensure that one of the n+1 output terminals outputs the second image data.

[0121] For example, if the value of S0 is 1, it means that the affine coupling layer F1 -1 Output second image data (see) Figure 8 In this case, the second image data can also be understood as the restored original image data.

[0122] For example, if S k A value of 1 indicates an affine coupling layer. Output second image data (see) Figure 9 In this case, the second image data can also be understood as the image data between the first image data and the restored original image data.

[0123] For example, if S nIf the value is 1, it means that the second image data is output directly. In this case, the image data can also be understood as the first image data.

[0124] It should be understood that the values ​​of the n+1 selection parameters affect the memory bandwidth and processing latency of the first module when processing the first image data. Specifically, if the value of a selection parameter closest to S0 is determined to be 1, then in actual operation, for the input first image data, most of the affine coupling layers of the first model will participate in the calculation, which will occupy more memory bandwidth and have a longer processing latency; if the value of a selection parameter closest to S0 is determined to be 1, then... n If one of the selection parameters is set to 1, then in actual operation, for the first input image data, a few affine coupling layers of the first model will participate in the calculation, which will occupy less memory bandwidth and have a shorter processing latency.

[0125] Meanwhile, the values ​​of the n+1 selection parameters also affect the image quality of subsequent processing. Specifically, if the value of a selection parameter closest to S0 is determined to be 1, then in actual operation, for the input first image data, most of the affine coupling layers of the first model will participate in the calculation. The more details in the restored second image data, the higher the image quality obtained by post-processing based on the second image data, and vice versa.

[0126] Therefore, when determining the values ​​of the n+1 selection parameters, the above-mentioned influencing factors can be comprehensively considered to select a more suitable output terminal to output the second image data.

[0127] Optionally, the values ​​of the n+1 selection parameters can be determined based on subjective experience, or they can be jointly determined with the parameters of the subsequent third module, as shown below.

[0128] Figure 10 This is a schematic diagram of the structure of the third model provided in an embodiment of this application. The third model is used to obtain third image data suitable for the second service based on the second image data output by the first model.

[0129] like Figure 10 As shown, the third model includes p cascaded affine coupling layers {T1,T2,...,T...} p The output of each of the p cascaded affine coupling layers serves as the input to the next affine coupling layer. The input of the first affine coupling layer T1 is the second image data, and the input of the p-th affine coupling layer T... p The output is the third image data, where p is a positive integer. The structure of each affine coupling layer can be found in [link to documentation]. Figure 5 The affine coupling layer F1 is shown.

[0130] Optionally, given that the values ​​of the n+1 selection parameters are determined, the model parameters of the third model can be trained based on the performance indicators of the second service. For example, if the second service is a CV-related service, the model parameters of the third model can be trained based on the performance indicators of the processing modules involved in the CV-related service; if the second service is an HV-related service, the model parameters of the third model can be trained based on subjective image quality evaluation indicators.

[0131] Optionally, when the values ​​of the n+1 selection parameters are uncertain, the values ​​of the n+1 selection parameters and the model parameters for synchronously training the third model can be determined based on the performance indicators of the second service and the bandwidth, memory, and latency indicators of the first model processing the first image data. In this case, during the training of the third model, a loss function can be predefined, and the third model can be trained based on this loss function. This embodiment of the application does not limit the specific form of this loss function. For example, the loss function can be as shown in formula (5):

[0132] Loss=||X1+X2+X3||2 (5)

[0133] Where X1 represents the performance of the second service, X2 represents the bandwidth and memory of the first model processing the first image data, and X3 represents the processing latency of the first model processing the first image data. When the second service is an HV-related service, X1 represents the subjective performance evaluation of image quality; when the second service is a CV-related service, X1 represents the performance of the processing modules involved in the CV-related service.

[0134] Before training, a target value (i.e., the training objective) is determined by combining the performance metrics of the second service with the bandwidth, memory, and latency metrics of the first model processing the first image data. During training, the values ​​of n+1 selected parameters and the model parameters of the third model are continuously adjusted based on the loss function value until the loss function value satisfies the aforementioned target value, thus completing the training of the third model.

[0135] Figure 11 This is a schematic diagram of the structure of a system provided in an embodiment of this application. For example... Figure 11 As shown. The system 1100 includes an image sensor 1110, an ISP 1120, an image processing device 1130, a first service processing module 1140, and a second service processing module 1150.

[0136] The image sensor 1110 is used to acquire raw image data.

[0137] The ISP 1120 is configured with a target ISP parameter group. The ISP 1120 processes the raw image data according to the target ISP parameter group and outputs the first image data for the first service to the first service processing module 1140.

[0138] The image processing device 1130 is used to execute the image processing method 400 described above to process the first image data to obtain the third image data applied to the second service, and to transmit the third image data to the second service processing module 1150.

[0139] The image processing device 1130 may include a first model 1131 and a third model 1133. The first model 1131 is used to process the first image data to obtain the second image data, and the third model 1133 is used to process the second image data to obtain the third image data. The first model 1131 can be obtained from the second model 1132, as detailed above.

[0140] In practice, multiple image processing devices can be configured on the back end corresponding to the target ISP parameter group of the ISP. These multiple image processing devices can simultaneously process the first image data output by the target ISP parameter group to obtain image data that can be applied to multiple services. Figure 12 This is a schematic diagram of another system structure provided in an embodiment of this application. For example... Figure 12 As shown. The system 1200 includes an image sensor 1210, an ISP 1220, N image processing devices 1230a-1230N (collectively referred to as 1230), a first service processing module 1240, and N different second service processing modules 1250a-1250N (collectively referred to as 1250).

[0141] The image sensor 1210 and ISP 1220 are described in the above description of image sensor 1110 and ISP 1120, and will not be repeated here.

[0142] N image processing devices 1230a-1230N are configured to: obtain N different second image data based on first image data, wherein the data size of each of the N different second image data is greater than or equal to the data size of the first image data and less than or equal to the data size of the original image data; and obtain N different third image data (i.e., third image data a to third image data N) based on the N different second image data, wherein the N different third image data are used for N different second services, and N is a positive integer greater than or equal to 1.

[0143] Among them, N image processing devices 1230a-1230N correspond one-to-one with N different second image data, N different second image data correspond one-to-one with N different third image data, and N different third image data correspond one-to-one with the second service processing modules 1250a-1250N.

[0144] In the embodiments of this application, the first image data can be processed in parallel by N image processing devices, and N different third image data can be output in parallel for N different second services, thereby increasing the number of services supported by the ISP.

[0145] Optionally, if the first service is an HV-related service, the N different second services can be N different CV-related services, or other N different HV-related services, which are different from the first service, or some are different CV-related services and the other part is other different HV-related services; if the first service is a CV-related service, the N different second services can be N different HV-related services, or other N different CV-related services, or some are different HV-related services and the other part is other different CV-related services, and this application does not limit this.

[0146] Optionally, each parameter group in the ISP can expand the applicable services in the manner described above, and this application does not limit this.

[0147] Figure 13 This is a schematic diagram of the structure of the image processing apparatus provided in an embodiment of this application. Figure 13 As shown, the device 1300 includes: an acquisition module 1310, a first processing module 1320, and a second processing module 1330.

[0148] The acquisition module 1310 is used to acquire first image data output by the ISP. The first image data is obtained by the ISP processing the original image data according to the target ISP parameter group. The first image data is used for the first service. The original image data is acquired by the image sensor.

[0149] The first processing module 1320 is used to obtain second image data based on the first image data, wherein the amount of the second image data is greater than or equal to the amount of the first image data and less than or equal to the amount of the original image data.

[0150] The second processing module 1330 is used to obtain third image data based on the second image data, and the third image data is used for the second service.

[0151] Optionally, the first processing module 1320 is used to reconstruct the first image data based on the first model to obtain the second image data. The first model may be obtained based on the second model, which may be trained based on the image data samples of the first service.

[0152] Optionally, the second model is used to simulate the output of the ISP based on the image data acquired by the image sensor.

[0153] Optionally, the second processing module 1330 is used to process the second image data based on the third model to obtain the third image data. The third model may be trained according to the requirements of the second service.

[0154] Optionally, the first model may include n cascaded computing units and n+1 output terminals. The output of each of the n cascaded computing units serves as the input of the next computing unit. The input of the first computing unit among the n computing units is the first image data. One of the n+1 output terminals outputs the second image data, where n is a positive integer.

[0155] Optionally, the first model determines the output terminal for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one with the n+1 output terminals.

[0156] Optionally, the value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output terminal that outputs the second image data is 1.

[0157] Figure 14 This is an exemplary block diagram of the hardware structure of the image processing apparatus provided in this application embodiment. Optionally, the apparatus 1400 may specifically be a computer device. The apparatus 1400 includes a memory 1410, a processor 1420, and a communication interface 1430. The memory 1410, processor 1420, and communication interface 1430 can be interconnected via a bus.

[0158] The memory 1410 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1410 may store a program, and when the program stored in the memory 1410 is executed by the processor 1420, the processor 1420 is used to execute the various steps of the image processing method 400 of the embodiments of this application.

[0159] The processor 1420 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, used to execute related programs to implement the image processing method 400 of the method embodiment of this application.

[0160] The processor 1420 can also be an integrated circuit chip with signal processing capabilities. In implementation, the image processing method 400 of this application can be completed through integrated logic circuits in the hardware of the processor 1420 or through software instructions.

[0161] The communication interface 1430 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 1400 and other devices or communication networks.

[0162] This application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the image processing method 400 described above.

[0163] This application also provides a computer program product containing instructions that, when executed on a computer, perform the above-described image processing method 400.

[0164] This application embodiment also provides a computing device, including: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform the above-described image processing method 400.

[0165] This application embodiment also provides a chip, the chip including a processor and a data interface, the processor reads instructions stored in the memory through the data interface to execute the above image processing method 400, or the chip executes the above image processing method 400 through hardware circuitry.

[0166] This application also provides an electronic device, including modules for performing the image processing method in any possible implementation of the image processing method 400 described above.

[0167] This application also provides a smart device, including the image processing apparatus 1300 and / or apparatus 1400 described above. This smart device can be a vehicle, a smart home device, a smart terminal, etc.

[0168] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0173] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: The first image data output by the image signal processor (ISP) is obtained. The first image data is obtained by the ISP processing raw image data according to the target ISP parameter group configured for the first service. The first image data is used for the first service. The raw image data is obtained by the image sensor. The second image data is obtained based on the first image data, wherein the amount of data in the second image data is greater than or equal to the amount of data in the first image data, and less than or equal to the amount of data in the original image data; The second image data is processed to obtain third image data, which is used for the second service; Wherein, the first service and the second service are respectively the human vision (HV) service and the computer vision (CV) service, or the first service and the second service are respectively the CV service and the HV service, or the first service and the second service are different services within the HV service, or the first service and the second service are different services within the CV service.

2. The method according to claim 1, characterized in that, The step of obtaining the second image data based on the first image data includes: The first image data is restored based on the first model to obtain the second image data. The first model is obtained based on the second model, which is trained based on the image data samples of the first service.

3. The method according to claim 2, characterized in that, The second model is used to simulate the output of the ISP based on the image data acquired by the image sensor.

4. The method according to claim 2 or 3, characterized in that, The process of processing the second image data to obtain the third image data includes: The second image data is processed based on a third model to obtain the third image data, wherein the third model is trained according to the requirements of the second service.

5. The method according to any one of claims 2 to 4, characterized in that, The first model includes n serially connected computing units and n+1 output terminals. The output of each of the n serially connected computing units serves as the input of the next computing unit. The input of the first computing unit in the n serially connected computing units is the first image data. One of the n+1 output terminals outputs the second image data, where n is a positive integer.

6. The method as described in claim 5, characterized in that, The first model determines the output terminal for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one with the n+1 output terminals.

7. The method as described in claim 6, characterized in that, The value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output terminal that outputs the second image data is 1.

8. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire first image data output by the image signal processor (ISP). The first image data is obtained by the ISP processing raw image data according to the target ISP parameter group configured for the first service. The first image data is used for the first service. The raw image data is acquired by the image sensor. A first processing module is configured to obtain second image data based on the first image data, wherein the amount of data in the second image data is greater than or equal to the amount of data in the first image data, and less than or equal to the amount of data in the original image data. The second processing module is used to process the second image data to obtain third image data, which is used for the second service. Wherein, the first service and the second service are respectively the human vision (HV) service and the computer vision (CV) service, or the first service and the second service are respectively the CV service and the HV service, or the first service and the second service are different services within the HV service, or the first service and the second service are different services within the CV service.

9. The apparatus according to claim 8, characterized in that, The first processing module is used for, The first image data is restored based on the first model to obtain the second image data. The first model is obtained based on the second model, which is trained based on the image data samples of the first service.

10. The apparatus according to claim 9, characterized in that, The second model is used to simulate the output of the ISP based on the image data acquired by the image sensor.

11. The apparatus according to claim 9 or 10, characterized in that, The second processing module is used for, The second image data is processed based on a third model to obtain the third image data, wherein the third model is trained according to the requirements of the second service.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The first model includes n serially connected computing units and n+1 output terminals. The output of each of the n serially connected computing units serves as the input of the next computing unit. The input of the first computing unit in the n serially connected computing units is the first image data. One of the n+1 output terminals outputs the second image data, where n is a positive integer.

13. The apparatus as claimed in claim 12, characterized in that, The first model determines the output terminal for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one with the n+1 output terminals.

14. The apparatus as claimed in claim 13, characterized in that, The value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output terminal that outputs the second image data is 1.

15. An image processing apparatus, characterized in that, The device includes an input / output interface, a processor, and a memory. The processor controls the input / output interface to send and receive signals or information. The memory stores a computer program. The processor retrieves and runs the computer program from the memory, causing the image processing device to perform the image processing method as described in any one of claims 1 to 7.

16. An image processing system, characterized in that, include: Image sensors are used to acquire raw image data; An image signal processor (ISP) is used to process the raw image data according to a target ISP parameter set configured for a first service to obtain first image data, which is used for the first service. N image processing devices are configured to: obtain N different second image data based on the first image data, wherein the data size of each of the N different second image data is greater than or equal to the data size of the first image data, and less than or equal to the data size of the original image data; Furthermore, the N different second image data are processed to obtain N different third image data, which are used for N different second services, where N is a positive integer greater than or equal to 1; Wherein, the first service and the second service are respectively HV service and CV service, or the first service and the second service are respectively CV service and HV service, or the first service and the second service are different services in the HV service, or the first service and the second service are different services in the CV service.

17. A smart device, characterized in that, The image processing apparatus includes any one of claims 8 to 15.

Images