Image processing method, device and system and intelligent equipment

Abstract

The invention provides an image processing method, device and system, and is suitable for the fields of intelligent vehicles, intelligent terminals, intelligent homes and the like. The method comprises the following steps: acquiring first image data output by an ISP (Internet Service Provider), the first image data being obtained by processing original image data by the ISP according to a target ISP parameter group, and the first image data being used for a first service; obtaining second image data according to the first image data, the data volume of the second image data being greater than or equal to the data volume of the first image data and less than or equal to the data volume of the original image data; and obtaining third image data according to the second image data, wherein the third image data is used for the second service. According to the scheme, the third image data which can be used for the second service is obtained by processing the first image data which is output by the ISP and is used for the first service, so that the number of services supported by the ISP is increased.

Description

Image processing method, device, system and intelligent device Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to an image processing method, apparatus, system and intelligent device. Background Art

[0002] The image signal processor (ISP) primarily processes raw image data captured by the front-end image sensor and transmits the processed image data to human vision (HV) or computer vision (CV) services. An ISP typically includes multiple data channels, each of which contains an ISP parameter group. This ISP parameter group must be configured based on the specific application.

[0003] Different services typically have different requirements for image quality indicators. An ISP parameter set configured for one service may not be sufficient for other services. For example, an ISP parameter set optimized for HV-related services based on subjective human evaluation may not be sufficient for CV-related services. Traditionally, different ISP parameter sets are configured on different ISP data channels, allowing the image data output by the ISP based on different ISP parameter sets to be used for different services. However, since one data channel can only correspond to one ISP parameter set, and the number of data channels is limited by the ISP's bandwidth, the services supported by the ISP are limited by the number of data channels.

[0004] Therefore, how to increase the number of services supported by ISPs is a technical problem that needs to be solved urgently.

[0005] Summary of the Invention

[0006] The present application provides an image processing method, apparatus, system, and intelligent device, which can increase the number of services supported by an ISP.

[0007] In a first aspect, an image processing method is provided, comprising: obtaining first image data output by an ISP, the first image data being obtained by the ISP processing original image data according to a target ISP parameter group, the first image data being used for a first service, the original image data being obtained by an image sensor; obtaining second image data based on the first image data, the data volume of the second image data being greater than or equal to the data volume of the first image data and less than or equal to the data volume of the original image data; and obtaining third image data based on the second image data, the third image data being used for a second service.

[0008] In the present application, on the one hand, third image data that can be used for a second service is obtained by processing the first image data output by the ISP for the first service, so as to achieve the expansion of the service corresponding to the target ISP parameter group in the ISP, thereby increasing the number of services supported by the ISP; on the other hand, in the process of processing the first image data to obtain the third image data, the second image data can be obtained based on the first image data first, and then the third image data can be obtained based on the second image data, wherein the data volume of the second image data can be greater than or equal to the data volume of the first image data and less than or equal to the data volume of the original image data, which means that the second image data can contain more image data information than the first image data, thereby improving the quality of the third image data.

[0009] Obtaining the second image data from the first image data can also be described as: restoring the first image data to obtain the second image data. Based on this, the second image data can be the first image data, image data in the restoration process, or restored original image data.

[0010] In combination with the first aspect, in certain implementations of the first aspect, obtaining the second image data based on the first image data includes: restoring the first image data based on a first model to obtain the second image data, the first model is obtained based on the second model, and the second model is trained based on the image data samples of the first business.

[0011] The image data sample of the first service may be an image data sample related to the first service. Specifically, the image data sample of the first service may include an original image data sample acquired from the image sensor and a first image data sample obtained by the ISP processing the original image data sample according to the target ISP parameter group. The first image data sample may be applied to the first service.

[0012] Specifically, the second model is trained based on raw image data samples acquired by the image sensor and first image data samples obtained by processing the raw image data samples by the ISP according to a target ISP parameter set. During the training process, the raw image data samples are used as input data for the training model, and the model parameters of the training model are continuously adjusted. The second model is obtained when the difference between the image data output by the training model and the first image data sample meets a preset value.

[0013] The second model is a reversible model, and the first model is obtained according to the inverse transformation model of the second model.

[0014] It should be understood that the aforementioned process of obtaining the second image data based on the first image data can be understood as the process of restoring the first image data. However, due to the relatively high complexity of the ISP algorithm, it is difficult to restore the first image data to obtain the second image data by directly performing an inverse operation based on the ISP model. Therefore, in this application, a second model different from the ISP is first trained using image data samples of the first service, and then the first model is derived from the second model to achieve restoration of the first image data, thereby reducing the complexity of restoring the first image data.

[0015] In combination with the first aspect, in some implementations of the first aspect, the second model is used to simulate the output of the ISP based on the image data acquired by the image sensor.

[0016] In other words, the second model can obtain the same output when the same data is input as the ISP. It should be understood that since the second model is different from the ISP itself, slight differences between the image data simulated by the second model and the image data output by the ISP are allowed in this application.

[0017] In combination with the first aspect, in certain implementations of the first aspect, obtaining the third image data based on the second image data includes: processing the second image data based on a third model to obtain the third image data, and the third model is trained according to the needs of the second business.

[0018] In the present application, the second image data can be processed based on the third model to obtain the third image data, and the third model is trained according to the needs of the second business, so that the quality of the third image data processed by the third model can meet the needs of the second business.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the first model includes n computing units connected in series and n+1 output terminals, the output of each computing unit in the n computing units connected in series serves as the input of the next computing unit, the input of the first computing unit in the n computing units is the first image data, and one output terminal in the n+1 output terminals outputs the second image data, where n is a positive integer.

[0020] The calculation unit may be a radiation coupling layer.

[0021] One of the n+1 output terminals is provided at the input of the first computing unit among the n computing units, and the other n output terminals are provided at the output of each computing unit among the n computing units.

[0022] The first model involved in the present application may include n computing units connected in series and n+1 output terminals, and one of the n+1 output terminals outputs the second image data, so that in the process of restoring the first image data, the output terminal for outputting the second image data can be selectively determined according to actual needs.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the first model determines the output end for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one to the n+1 output ends.

[0024] In combination with the first aspect, in certain implementations of the first aspect, the value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output end that outputs the second image data is 1.

[0025] The value of the selection parameter is 0 for no output, and 1 for output.

[0026] In a second aspect, an image processing device is provided, including: an acquisition module for acquiring first image data output by an ISP, the first image data being obtained by the ISP processing original image data according to a target ISP parameter group, the first image data being used for a first business, and the original image data being acquired by an image sensor; a first processing module for obtaining second image data based on the first image data, the data volume of the second image data being greater than or equal to the data volume of the first image data and less than or equal to the data volume of the original image data; a second processing module for obtaining third image data based on the second image data, the third image data being used for a second business.

[0027] In combination with the second aspect, in certain implementations of the second aspect, the first processing module is used to restore the first image data based on the first model to obtain the second image data, the first model is obtained based on the second model, and the second model is trained based on the image data samples of the first business.

[0028] In combination with the second aspect, in some implementations of the second aspect, the second model is used to simulate the output of the ISP based on the image data acquired by the image sensor.

[0029] In combination with the second aspect, in some implementations of the second aspect, the second processing module is used to process the second image data based on a third model to obtain the third image data, and the third model is trained according to the needs of the second business.

[0030] In combination with the second aspect, in certain implementations of the second aspect, the first model includes n computing units connected in series and n+1 output terminals, the output of each computing unit in the n computing units connected in series serves as the input of the next computing unit, the input of the first computing unit in the n computing units is the first image data, and one output terminal in the n+1 output terminals outputs the second image data, where n is a positive integer.

[0031] In combination with the second aspect, in certain implementations of the second aspect, the first model determines the output end for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one to the n+1 output ends.

[0032] In combination with the second aspect, in certain implementations of the second aspect, the value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output end that outputs the second image data is 1.

[0033] In a third aspect, an image processing device is provided, comprising an input / output interface, a processor, and a memory, wherein the processor is used to control the input / output interface to send and receive signals or information, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the image processing device performs the image processing method in the first aspect or any possible implementation of the first aspect.

[0034] In a fourth aspect, an image processing system is provided, comprising: an image sensor for acquiring raw image data; an ISP for processing the raw image data according to a target ISP parameter group to obtain first image data, wherein the first image data is used for a first business; N image processing devices for: obtaining N different second image data according to the first image data, wherein the data amount of each second image data in the N different second image data is greater than or equal to the data amount of the first image data, and less than or equal to the data amount of the original image data; and obtaining N different third image data according to the N different second image data, wherein the N different third image data are used for N different second businesses, where N is a positive integer greater than or equal to 1.

[0035] Among them, the N image processing devices correspond one-to-one to N different second image data, the N different second image data correspond one-to-one to N different third image data, and the N different third image data correspond one-to-one to N different second services.

[0036] In the present application, the first image data can be processed in parallel by N image processing devices, and N different third image data can be output in parallel for N different second services, thereby increasing the number of services supported by the ISP.

[0037] In a fifth aspect, a computer-readable medium is provided, characterized in that the computer-readable medium stores program code, and when the program code is run on a computer, the computer executes the image processing method in the first aspect or any possible implementation of the first aspect.

[0038] In a sixth aspect, a computer program product is provided, comprising instructions. When the instructions are run on a computer, the image processing method in the first aspect or any possible implementation of the first aspect is executed.

[0039] In the seventh aspect, a computing device is provided, comprising: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform an image processing method as in the first aspect or any possible implementation of the first aspect.

[0040] In an eighth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes an image processing method as in the first aspect or any possible implementation of the first aspect.

[0041] Optionally, as an implementation method, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored on the memory. When the instructions are executed, the processor is used to execute the image processing method in the first aspect or any possible implementation method of the first aspect.

[0042] In a ninth aspect, an electronic device is provided, characterized in that it includes various modules for executing the image processing method in the first aspect or any possible implementation of the first aspect.

[0043] In the tenth aspect, an intelligent driving device is provided, characterized in that it includes an image processing device as in the second aspect or any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a schematic diagram of a traditional system architecture for image processing.

[0045] FIG2 is a schematic diagram of an ISP data channel provided in an embodiment of the present application.

[0046] FIG3 is a schematic diagram of a configuration scheme of an ISP data channel provided in an embodiment of the present application.

[0047] FIG4 is a schematic diagram of an image processing method provided in an embodiment of the present application.

[0048] FIG5 is a schematic structural diagram of the second model provided in an embodiment of the present application.

[0049] FIG6 is a schematic diagram of the training process of the second model provided in an embodiment of the present application.

[0050] FIG7 is a schematic structural diagram of the first model provided in an embodiment of the present application.

[0051] FIG8 is a schematic structural diagram of the first model provided in an embodiment of the present application.

[0052] FIG9 is a schematic structural diagram of the first model provided in an embodiment of the present application.

[0053] FIG10 is a schematic structural diagram of the third model provided in an embodiment of the present application.

[0054] FIG11 is a schematic structural diagram of a system provided in an embodiment of the present application.

[0055] FIG12 is a schematic structural diagram of another system provided in an embodiment of the present application.

[0056] FIG13 is a schematic diagram of the structure of the image processing device provided in an embodiment of the present application.

[0057] FIG14 is an exemplary block diagram of the hardware structure of the image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0059] The present application solution can be applied to smart vehicles, smart terminals, smart homes, and other fields. For example, it can be applied to smart phones, desktop computers, laptops, tablet computers, wearable devices, home robots, or smart driving devices, etc., where ISP can be used for image processing.

[0060] Among them, intelligent driving devices can include road vehicles, water vehicles, air vehicles, industrial equipment, agricultural equipment, or entertainment equipment, etc. For example, the intelligent driving device can be a vehicle, which is a vehicle in a broad sense, and can be a vehicle (such as a commercial vehicle, a passenger car, a motorcycle, a flying car, a train, etc.), an industrial vehicle (such as a forklift, a trailer, a tractor, etc.), an engineering vehicle (such as an excavator, a bulldozer, a crane, etc.), agricultural equipment (such as a mower, a harvester, etc.), amusement equipment, a toy vehicle, etc. The embodiment of this application does not specifically limit the type of vehicle. For another example, the intelligent driving device can be a vehicle such as an airplane or a ship.

[0061] To facilitate understanding, the following first briefly introduces the background technology involved in the embodiments of the present application in conjunction with Figures 1 to 3.

[0062] FIG1 is a schematic diagram of a conventional system architecture for image processing. As shown in FIG1 , the system architecture 100 includes an image sensor 110 , an ISP 120 , and a service processing module 130 .

[0063] The image sensor 110 is used to sense the external environment, acquire raw image data, and then transmit the raw image data to the ISP 120 via a serializer and a deserializer. The image sensor 110 may be, for example, a camera. The ISP 120 processes the raw image data acquired by the image sensor 110, performing various processing steps, such as black level compensation, bad pixel correction, color interpolation, Bayer noise removal, white balance correction, color restoration correction, gamma correction, color noise removal and edge enhancement in the color space, and color and contrast enhancement. The ISP then transmits the processed image data to the service processing module 130 for use.

[0064] The service processing module 130 is used to process services. There can be one or more service processing modules 130. Services can be HV-related services or CV-related services. Taking the automotive field as an example, CV-related services may include target detection, lane detection, traffic light recognition, and semantic segmentation. HV-related services may include electronic rearview mirror data processing, surround view stitching, and rearview camera data processing.

[0065] It should be understood that transmitting the processed image data by ISP 120 to the service processing module 130 for use means transmitting the processed image data to the service processing module 130 corresponding to the service. For example, if the service is lane detection, the processed image data by ISP 120 is transmitted to the service processing module 130a related to lane detection. If the service is electronic rearview mirror data processing, the processed image data by ISP 120 is transmitted to the service processing module 130b related to the electronic rearview mirror.

[0066] ISP usually includes multiple data channels, and one ISP parameter group can be configured in each data channel. Figure 2 is a schematic diagram of the ISP data channel provided in an embodiment of the present application. As shown in Figure 2, the ISP includes L data channels, and L data channels can be configured with L sets of ISP parameter groups. It should be understood that each ISP parameter group in the ISP can output one image data, and each ISP parameter group needs to be configured in combination with the actual application business. Usually, different businesses have different requirements for image quality indicators. The ISP parameter group configured for a certain business may not be able to meet the needs of other businesses. For example, the ISP parameter group suitable for HV-related businesses obtained by tuning the indicators based on human subjective evaluation may not be able to meet the needs of CV-related businesses.

[0067] Traditional solutions typically configure different ISP parameter groups on different ISP data channels, allowing the image data output by the ISP based on these different ISP parameter groups to be used for different services. As shown in Figure 3, ISP parameter groups suitable for CV-related services can be configured on some data channels, while ISP parameter groups suitable for HV-related services can be configured on other data channels, allowing the image data output by the ISP to be used for different services. However, currently, since each data channel corresponds to only one ISP parameter group and the number of data channels is limited by the ISP's bandwidth, the services supported by the ISP are limited by the number of data channels.

[0068] Furthermore, an ISP's data channel typically only connects to one image sensor, and a particular service might require multiple image sensors simultaneously to acquire image data from different perspectives. These multiple image sensors correspond to multiple ISP data channels. Therefore, adding additional services requires occupying the data channel and image sensors corresponding to the current service, reducing the number of image sensors available for the current service and thus impacting the acquisition of the image information required by that service.

[0069] Based on this, the present application provides an image processing method, which processes the image data suitable for one business output by the ISP to obtain image data suitable for another business, so as to achieve the expansion of the business corresponding to an ISP parameter group in the ISP, thereby increasing the number of businesses supported by the ISP and ensuring that the number of image sensors corresponding to each business is not affected.

[0070] Figure 4 is a schematic diagram of an image processing method provided by an embodiment of the present application. As shown in Figure 4, the method 400 includes steps S410 to S430, which are described in detail below.

[0071] S410: Acquire first image data output by an ISP, where the first image data is used for a first service.

[0072] The first image data is obtained by the ISP processing raw image data according to a target ISP parameter group, and the raw image data is acquired by an image sensor.

[0073] The target ISP parameter group is an ISP parameter group on a data channel in the ISP. It should be understood that this application solution mainly uses the target ISP parameter group as an example. In actual operation, this application solution is also applicable to ISP parameter groups on one or more other data channels in the ISP.

[0074] S420: Obtain second image data according to the first image data.

[0075] The data size of the second image data is greater than or equal to the data size of the first image data, and less than or equal to the data size of the original image data.

[0076] It should be understood that obtaining the second image data based on the first image data can also be understood as performing a restoration process on the first image data to obtain the second image data. Based on this, the second image data can be the first image data, the restored original image data, or image data between the first image data and the restored original image data. It should be understood that if the second image data is image data between the first image data and the restored original image data, it will have a larger data volume than the first image data and can contain more image data information. This can also reduce the computational effort, memory usage, and bandwidth usage in the process of obtaining the second image data based on the first image data compared to the original image data.

[0077] Optionally, the first image data can be restored based on the first model to obtain the second image data. The first model can include n computing units connected in series and n+1 output terminals, where the output of each computing unit in the n computing units connected in series serves as the input of the next computing unit, the input of the first computing unit in the n computing units is the first image data, and one of the n+1 output terminals outputs the second image data, where n is a positive integer.

[0078] The calculation unit may be a radiation coupling layer.

[0079] One of the n+1 output terminals is provided at the input of the first computing unit among the n computing units, and the other n output terminals are provided at the output of each of the n computing units. Optionally, the output terminal for outputting the second image data among the n+1 output terminals can be determined based on the needs of the second service. For details, see below.

[0080] The first model can determine the output terminal for outputting the second image data using n+1 selection parameters, where the n+1 selection parameters correspond one-to-one to the n+1 output terminals. In other words, the n+1 selection parameters can be used to control n of the n+1 output terminals to not output, while one output terminal can output.

[0081] Exemplarily, the value of each of the n+1 selection parameters may be 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output terminal that outputs the second image data is 1. A value of 0 for the selection parameter indicates no output, and a value of 1 indicates output.

[0082] As an implementation, the first model can be derived from a second model, which is used to simulate the output of the ISP based on image data acquired by the image sensor. In other words, the second model can produce the same output when fed the same data as the ISP. It should be understood that because the second model is different from the ISP itself, slight differences between the image data simulated by the second model and the image data output by the ISP are permitted in the embodiments of this application.

[0083] The second model may include n computing units connected in series, the output of each computing unit in the n computing units connected in series serves as the input of the next computing unit, the input of the first computing unit in the n computing units serves as the input of the second model, and the output of the nth computing unit serves as the output of the second model.

[0084] The second model is trained based on image data samples of the first service. The image data samples of the first service may be image data samples related to the first service. Specifically, the image data samples of the first service may include original image data samples acquired from the image sensor and first image data samples obtained by the ISP processing the original image data samples according to the target ISP parameter set. The first image data samples may be applied to the first service.

[0085] Specifically, the second model is trained based on raw image data samples acquired by the image sensor and first image data samples obtained by the ISP processing the raw image data samples according to a target ISP parameter set. During the training process, the raw image data samples are used as input data for training the second model, and the model parameters of the trained second model are continuously adjusted. The second model is obtained when the difference between the image data output by the trained second model and the first image data samples meets a preset value.

[0086] The second model is a reversible model, and the first model can be obtained based on the second model, or described as follows: the n serially connected computing units of the first model are obtained by performing inverse transformation processing on the second model.

[0087] In this application, obtaining the second image data based on the first image data can be understood as the process of restoring the first image data. However, due to the high complexity of the ISP algorithm, it is difficult to restore the first image data to obtain the second image data by directly performing an inverse operation based on the ISP model. Therefore, in an embodiment of this application, a second model different from the ISP is first trained using image data samples of the first service, and then the first model is derived from the second model to achieve restoration of the first image data, thereby reducing the complexity of restoring the first image data.

[0088] The process of constructing the second model and obtaining the first model based on the second model can be found below.

[0089] As another implementation, the first model can also be trained directly based on the image data samples of the first service. Specifically, during the training process, the first image data samples are used as input data for training the first model, and model parameters of the first model are continuously adjusted. The first model is obtained when the difference between the image data output by the trained first model and the original image data samples meets a preset value.

[0090] S430: Obtain third image data according to the second image data, where the third image data is used for a second service.

[0091] Optionally, the second image data may be processed based on a third model to obtain third image data, where the third model is trained according to requirements of the second service.

[0092] The requirements of the second business may include performance requirements, resource requirements, etc. Among them, the performance requirements refer to the performance index requirements of the processing modules involved in the second business. For example, if the second business is a CV-related business, the performance requirements may refer to the performance index requirements of the processing modules involved in the CV-related business (for example: perception networks such as deep neural networks (DNN)) when taking the third image data as input and outputting the image data, such as accuracy, precision, recall rate, etc. If the second business is an HV-related business, the performance requirements may refer to the subjective evaluation index requirements of the processing modules involved in the HV-related business when taking the third image data as input and outputting the image data quality. Resource requirements may include index requirements such as memory bandwidth and data processing delay, and the index requirements such as memory bandwidth and data processing delay refer to the index requirements such as memory bandwidth and data processing delay occupied by the process of obtaining the third image data applied to the second business according to the solution of this application.

[0093] In an embodiment of the present application, the second image data can be processed based on a third model to obtain third image data, and the third model is trained according to the needs of the second business, so that the quality of the third image data processed by the third model can meet the needs of the second business.

[0094] Optionally, the output end for outputting the second image data and the training of the third model can be determined based on the performance indicators of the second business; or, the output end for outputting the second image data and the training of the third model can be determined based on the bandwidth memory indicators and / or delay indicators of the first model processing the first image data, and the performance indicators of the second business processing.

[0095] The third model may include p computing units connected in series, the output of each computing unit in the p computing units connected in series serves as the input of the next computing unit, the input of the first computing unit in the p computing units is the second image data, the output of the pth computing unit is the third image data, and p is a positive integer.

[0096] It should be understood that the target parameter group in the ISP typically includes a large number of parameters. In actual operation, it is also possible to use some of the parameters in the target parameter group to obtain the first image data, while the functions required to implement the remaining parameters can be implemented through processing by the third model. This means that during the training process of the third model, it is also necessary to train based on the functions required to implement the aforementioned remaining parameters in order to implement these functions through the third model. For example, white balance correction can be performed without performing the image processing during the ISP, but instead can be implemented during the subsequent processing of the third model.

[0097] The businesses involved in this application may include HV-related businesses and CV-related businesses, etc. The introduction of HV-related businesses and CV-related businesses can be found above.

[0098] Based on this, the above-mentioned first business can be one of the HV-related businesses, and the second business can be one of the CV-related businesses; or the first business can be one of the CV-related businesses, and the second business can be one of the HV-related businesses; or, the first business can be the first business among the HV-related businesses, and the second business can be the second business among the HV-related businesses; or, the first business can be the first business among the CV-related businesses, and the second business can be the second business among the CV-related businesses.

[0099] In an embodiment of the present application, on the one hand, third image data that can be used for a second service is obtained by processing first image data output by the ISP for a first service, so as to achieve expansion of the service corresponding to the target ISP parameter group in the ISP, thereby increasing the number of services supported by the ISP; on the other hand, in the process of processing the first image data to obtain the third image data, the second image data can be obtained first based on the first image data, and then the third image data can be obtained based on the second image data, wherein the data volume of the second image data can be greater than or equal to the data volume of the first image data and less than or equal to the data volume of the original image data, which means that the second image data can contain more image data information than the first image data, thereby improving the quality of the third image data.

[0100] Taking the calculation unit as an affine coupling layer as an example, the model construction of the first model, the second model and the third model are respectively described in detail in combination with Figures 5 to 10.

[0101] Figure 5 is a schematic diagram of the structure of the second model provided in an embodiment of the present application. The second model can simulate the output of the ISP based on the image data obtained by the image sensor. In the embodiment of the present application, the construction of the second model is mainly to obtain the first model.

[0102] As shown in FIG5 , the second model includes n series-connected affine coupling layers {F1, F2, ..., F k ,...,F n}, the output of each affine coupling layer in the n series affine coupling layers is used as the input of the next affine coupling layer, the input of the first affine coupling layer F1 in the n affine coupling layers is the input of the second model, and the nth affine coupling layer F n The output of is the output of the second model. The affine coupling layer has a reversible characteristic.

[0103] The specific structure of each radiation coupling layer can be referred to the affine coupling layer F1 shown in FIG5 , and the specific structure can be expressed by the following formula (1):

[0104]

[0105] Among them, x a and x b is the input data of the affine coupling layer. Taking the affine coupling layer F1 as an example, x a and x b is the two parts of image data divided according to the first image data; m a and m b is the output data of the affine coupling layer; the symbol ⊙ represents the multiplication of each element, s(x a ) and t(x a ) is a mapping function. It should be understood that s(x a ) and t(x a ) can be a linear or nonlinear function, without limitation. For example, s(x a ) can be expressed as formula (2):

[0106] s(x a )=αx a +β (2)

[0107] Among them, α and β are the parameters that can be trained in the second model.

[0108] It should be understood that the above formula (2) is only used as s(x a ) is an example, in actual operation, other forms can also be used without limitation. In addition, t(x a ) is expressed in the same way as s(x a )similar.

[0109] Based on the second model structure created above, the model parameters of the second model are trained in combination with the image data samples of the first service. The image data samples of the first service may include raw image data samples acquired from the image sensor and first image data samples obtained by processing the raw image data samples by the ISP, and the first image data samples may be applied to the first service.

[0110] The following is an exemplary description of the training process of the second model with reference to Figure 6. As shown in Figure 6, the original image data sample can be simultaneously input into the ISP and the second model. Then, the first image data sample output by the ISP and the image data output by the second model are compared, and the difference between the two output data is calculated. The model parameters of the second model are continuously adjusted based on the difference until the difference between the output of the ISP and the output of the second model meets the preset value, thus completing the training of the second model.

[0111] Alternatively, a loss function may be defined in advance and the second model may be trained based on the loss function. The embodiment of the present application does not limit the specific form of the loss function. For example, the loss function may be as shown in formula (3):

[0112] Loss=||IMG1-IMG2||2 (3)

[0113] Among them, IMG1 is the output of ISP, and IMG2 is the output of the second model.

[0114] It should be understood that the higher the function value of the loss function, the greater the difference between the output of the ISP and the output of the second model. During the training process, the function value of the loss function can be calculated based on the output of the ISP and the output of the second model, and the model parameters of the second model can be continuously adjusted according to the loss function value until the function value of the loss function meets the preset value, that is, the training of the second model is completed.

[0115] After determining the second model, the second model can be inversely transformed based on the reversible characteristics of the affine coupling layer to obtain the first model, and then the first image data output by the ISP can be reversely restored based on the first model. The first model is described below with reference to Figures 7 to 9.

[0116] 7 is a schematic diagram of the structure of the first model provided in an embodiment of the present application. The first model is mainly used to reversely restore the first image data output by the ISP to obtain more image data information.

[0117] As shown in FIG7 , the first model includes n series-connected affine coupling layers and n+1 output terminals, the output of each affine coupling layer in the n series-connected affine coupling layers serves as the input of the next affine coupling layer, and the first affine coupling layer in the n affine coupling layers The input is the first image data, and one of the n+1 output ends outputs the second image data, where n is a positive integer.

[0118] Among them, n series affine coupling layers It is achieved by connecting n series affine coupling layers {F1, F2, ..., F k ,...,F n} Obtained by inverting layer by layer.

[0119] Take the nth affine coupling layer of the first model For example, the affine coupling layer is obtained by inverting the first affine coupling layer F1 in the second model. Based on this, the affine coupling layer The specific structure of can be expressed by the following formula (4):

[0120]

[0121] For example, referring to FIG7 , n+1 selection parameters {S n ,S n-1 ,...,S k ,...,S1,S0} determines which output end of the n+1 output ends outputs the second image data. Wherein, the value of S is 0 or 1, 0 represents no output, 1 represents output, and S0+S1+...+S n =1, which means that one of the n+1 selection parameters is 1 and the others are 0, so as to ensure that one of the n+1 output terminals outputs the second image data.

[0122] For example, if the value of S0 is 1, it means that the affine coupling layer The second image data is output (see FIG. 8 ). In this case, the second image data can also be understood as restored original image data.

[0123] For example, if S k A value of 1 means that the affine coupling layer The second image data is output (see FIG. 9 ). In this case, the second image data can also be understood as image data between the first image data and the restored original image data.

[0124] For example, if S n If the value of is 1, it means that the second image data is directly output. In this case, the image data can also be understood as the first image data.

[0125] It should be understood that the parameter values ​​of the n+1 selection parameters affect the memory bandwidth and processing delay occupied by the first module when processing the first image data. Specifically, if the value of a selection parameter close to S0 is determined to be 1, then in actual operation, for the input first image data, most of the affine coupling layers of the first model will participate in the calculation, which will occupy more memory bandwidth and have a longer processing delay; if the value of a selection parameter close to S0 is determined to be 1, then in actual operation, for the input first image data, most of the affine coupling layers of the first model will participate in the calculation, which will occupy more memory bandwidth and have a longer processing delay; n If the value of a selection parameter is 1, in actual operation, for the input first image data, a few affine coupling layers of the first model will participate in the calculation, which will occupy less memory bandwidth and have a shorter processing delay.

[0126] At the same time, the values ​​of the n+1 selection parameters also affect the quality of the image after subsequent processing. Specifically, if a selection parameter close to S0 is set to 1, in actual operation, most of the affine coupling layers of the first model will participate in the calculation of the input first image data. The more details in the restored second image data, the higher the quality of the image obtained after post-processing based on this second image data, and vice versa.

[0127] Therefore, when determining the parameter values ​​of the n+1 selection parameters, the above-mentioned influencing factors may be comprehensively considered to select a more appropriate output terminal to output the second image data.

[0128] Optionally, the parameter values ​​of the n+1 selection parameters may be determined based on subjective experience, or may be determined jointly with the parameters of a subsequent third module, as described below.

[0129] 10 is a schematic diagram of the structure of a third model provided in an embodiment of the present application. The third model is used to obtain third image data suitable for a second service based on the second image data output by the first model.

[0130] As shown in FIG10 , the third model includes p series-connected affine coupling layers {T1, T2, ..., T p}, the output of each affine coupling layer in the p series affine coupling layers is used as the input of the next affine coupling layer, the input of the first affine coupling layer T1 in the p affine coupling layers is the second image data, and the input of the pth affine coupling layer T p The output of is the third image data, and p is a positive integer. The structure of each affine coupling layer can refer to the affine coupling layer F1 shown in FIG5 .

[0131] Optionally, when the values ​​of the n+1 selection parameters are determined, the model parameters of the third model can be trained based on the performance indicators of the second service. For example, if the second service is a CV-related service, the model parameters of the third model can be trained based on the performance indicators of the processing modules involved in the CV-related service; if the second service is an HV-related service, the model parameters of the third model can be trained based on subjective image quality evaluation indicators.

[0132] Optionally, when the values ​​of the n+1 selection parameters are uncertain, the values ​​of the n+1 selection parameters and the model parameters of the third model can be determined based on the performance indicators of the second service and the bandwidth memory indicators and delay indicators of the first model for processing the first image data. Based on this situation, in the process of training the third model, a loss function can be defined in advance, and the third model can be trained based on the loss function. The embodiment of the present application does not limit the specific form of the loss function. For example, the loss function can be shown as formula (5):

[0133] Loss=||X1+X2+X3||2 (5)

[0134] Where X1 represents the performance of the second service, X2 represents the bandwidth and memory used by the first model to process the first image data, and X3 represents the processing latency of the first model to process the first image data. When the second service is an HV-related service, X1 represents the subjective performance evaluation of image quality; when the second service is a CV-related service, X1 represents the performance of the processing modules involved in the CV-related service.

[0135] Prior to training, a target value (also known as a training objective) is determined by comprehensively considering the performance indicators of the second service and the bandwidth, memory, and latency indicators of the first model for processing the first image data. During training, the values ​​of the n+1 selected parameters and the model parameters of the third model are continuously adjusted based on the loss function value until the loss function value meets the aforementioned target value, thus completing the training of the third model.

[0136] FIG11 is a schematic diagram of the structure of a system provided by an embodiment of the present application. As shown in FIG11 , the system 1100 includes an image sensor 1110 , an ISP 1120 , an image processing device 1130 , a first service processing module 1140 , and a second service processing module 1150 .

[0137] The image sensor 1110 is used to obtain original image data.

[0138] The ISP 1120 is configured with a target ISP parameter group. The ISP 1120 processes the original image data according to the target ISP parameter group and outputs the first image data for the first service to the first service processing module 1140 .

[0139] The image processing device 1130 is configured to execute the above-mentioned image processing method 400 to process the first image data to obtain third image data applied to the second service, and transmit the third image data to the second service processing module 1150 .

[0140] The image processing device 1130 may include a first model 1131 and a third model 1133. The first model 1131 is used to process the first image data to obtain the second image data, and the third model 1133 is used to process the second image data to obtain the third image data. The first model 1131 can be obtained based on the second model 1132. For details, please refer to the above description.

[0141] In actual operation, multiple image processing devices can also be configured at the back end corresponding to the target ISP parameter group of the ISP. The multiple image processing devices can simultaneously process the first image data output by the target ISP parameter group to obtain image data that can be applied to multiple businesses. Figure 12 is a structural diagram of another system provided by an embodiment of the present application. As shown in Figure 12. The system 1200 includes an image sensor 1210, an ISP 1220, N image processing devices 1230a-1230N (collectively referred to as 1230), a first business processing module 1240 and N different second business processing modules 1250a-1250N (collectively referred to as 1250).

[0142] Among them, the image sensor 1210 and the ISP 1220 can refer to the introduction of the above-mentioned image sensor 1110 and the ISP 1120, and will not be repeated here.

[0143] N image processing devices 1230a-1230N are used to: obtain N different second image data based on the first image data, the data amount of each second image data in the N different second image data is greater than or equal to the data amount of the first image data, and less than or equal to the data amount of the original image data; and obtain N different third image data (i.e., third image data a to third image data N) based on the N different second image data, the N different third image data are used for N different second services, and N is a positive integer greater than or equal to 1.

[0144] Among them, the N image processing devices 1230a-1230N correspond one-to-one to N different second image data, the N different second image data correspond one-to-one to N different third image data, and the N different third image data correspond one-to-one to the second business processing modules 1250a-1250N.

[0145] In an embodiment of the present application, the first image data can be processed in parallel by N image processing devices, and N different third image data can be output in parallel for N different second services, thereby increasing the number of services supported by the ISP.

[0146] Optionally, if the first business is an HV-related business, the N different second businesses can be N different CV-related businesses, or N other different HV-related businesses, the other N different businesses are different from the first business, or one part is different CV-related businesses, and the other part is other different HV-related businesses; if the first business is a CV-related business, the N different second businesses can be N different HV-related businesses, or N other different CV-related businesses, or one part is different HV-related businesses, and the other part is other different CV-related businesses. This application does not limit this.

[0147] Optionally, each parameter group in the ISP can extend applicable services in the above manner, which is not limited in this application.

[0148] FIG13 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present application. As shown in FIG13 , the device 1300 includes: an acquisition module 1310 , a first processing module 1320 , and a second processing module 1330 .

[0149] The acquisition module 1310 is used to acquire first image data output by the ISP. The first image data is obtained by the ISP processing the original image data according to the target ISP parameter group. The first image data is used for the first business. The original image data is acquired by the image sensor.

[0150] The first processing module 1320 is configured to obtain second image data according to the first image data. The data size of the second image data is greater than or equal to the data size of the first image data and less than or equal to the data size of the original image data.

[0151] The second processing module 1330 is configured to obtain third image data according to the second image data, where the third image data is used for a second service.

[0152] Optionally, the first processing module 1320 is configured to restore the first image data based on a first model to obtain second image data. The first model may be obtained based on a second model, and the second model may be trained based on image data samples of the first service.

[0153] Optionally, the second model is used to simulate the output of the ISP based on image data acquired by the image sensor.

[0154] Optionally, the second processing module 1330 is configured to process the second image data based on a third model to obtain third image data. The third model may be trained according to requirements of the second service.

[0155] Optionally, the first model may include n computing units connected in series and n+1 output terminals, the output of each computing unit in the n computing units connected in series serves as the input of the next computing unit, the input of the first computing unit in the n computing units is the first image data, and one output terminal in the n+1 output terminals outputs the second image data, where n is a positive integer.

[0156] Optionally, the first model determines the output end for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one to the n+1 output ends.

[0157] Optionally, the value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output end that outputs the second image data is 1.

[0158] FIG14 is a block diagram illustrating the hardware structure of an image processing apparatus provided in an embodiment of the present application. Optionally, apparatus 1400 may be a computer device. Apparatus 1400 includes a memory 1410, a processor 1420, and a communication interface 1430. Memory 1410, processor 1420, and communication interface 1430 may be interconnected via a bus.

[0159] The memory 1410 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1410 may store programs. When the program stored in the memory 1410 is executed by the processor 1420, the processor 1420 is configured to perform the various steps of the image processing method 400 of the embodiment of the present application.

[0160] The processor 1420 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the image processing method 400 of the method embodiment of the present application.

[0161] The processor 1420 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, the image processing method 400 of the present application may be completed by hardware integrated logic circuits in the processor 1420 or software instructions.

[0162] The communication interface 1430 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 1400 and other devices or a communication network.

[0163] An embodiment of the present application further provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the above-mentioned image processing method 400.

[0164] The embodiment of the present application further provides a computer program product, comprising instructions. When the instructions are executed on a computer, the above-mentioned image processing method 400 is performed.

[0165] An embodiment of the present application further provides a computing device, comprising: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is configured to read and execute instructions in the memory to perform the above-mentioned image processing method 400.

[0166] An embodiment of the present application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to execute the above-mentioned image processing method 400, or the chip executes the above-mentioned image processing method 400 through a hardware circuit.

[0167] An embodiment of the present application further provides an electronic device, comprising modules for executing the image processing method in any possible implementation of the above-mentioned image processing method 400.

[0168] The present application also provides an intelligent device including the image processing apparatus 1300 and / or apparatus 1400 described above. The intelligent device may be a vehicle, a smart home appliance, a smart terminal, or the like.

[0169] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0172] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0173] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0174] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire first image data output by an image signal processor (ISP), where the first image data is obtained by the ISP processing raw image data according to a target ISP parameter group, the first image data is used for a first service, and the raw image data is acquired by an image sensor; obtaining second image data according to the first image data, wherein the amount of the second image data is greater than or equal to the amount of the first image data and less than or equal to the amount of the original image data; Third image data is obtained according to the second image data, and the third image data is used for a second service.

2. The method according to claim 1, characterized in that The obtaining of the second image data according to the first image data comprises: The first image data is restored based on a first model to obtain the second image data, the first model is obtained based on a second model, and the second model is obtained by training based on image data samples of the first business.

3. The method according to claim 2, characterized in that The second model is used to simulate the output of the ISP according to the image data acquired by the image sensor.

4. The method according to claim 2 or 3, characterized in that The obtaining of the third image data according to the second image data comprises: The second image data is processed based on a third model to obtain the third image data, where the third model is trained according to requirements of the second business.

5. The method according to any one of claims 2 to 4, characterized in that The first model includes n computing units connected in series and n+1 output terminals, the output of each computing unit in the n computing units connected in series serves as the input of the next computing unit, the input of the first computing unit in the n computing units is the first image data, and one output terminal in the n+1 output terminals outputs the second image data, where n is a positive integer.

6. The method according to claim 5, wherein The first model determines an output terminal for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one to the n+1 output terminals.

7. The method according to claim 6, wherein The value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output terminal that outputs the second image data is 1.

8. An image processing device, characterized in that: include: an acquisition module, configured to acquire first image data output by an image signal processor (ISP), wherein the first image data is obtained by the ISP processing raw image data according to a target ISP parameter group, the first image data is used for a first service, and the raw image data is acquired by an image sensor; a first processing module, configured to obtain second image data based on the first image data, wherein the amount of the second image data is greater than or equal to the amount of the first image data and less than or equal to the amount of the original image data; The second processing module is configured to obtain third image data according to the second image data, where the third image data is used for a second service.

9. The device according to claim 8, characterized in that The first processing module is used to: The first image data is restored based on a first model to obtain the second image data, the first model is obtained based on a second model, and the second model is obtained by training based on image data samples of the first business.

10. The device according to claim 9, characterized in that The second model is used to simulate the output of the ISP according to the image data acquired by the image sensor.

11. The device according to claim 9 or 10, characterized in that The second processing module is used to: The second image data is processed based on a third model to obtain the third image data, where the third model is trained according to requirements of the second business.

12. The device according to any one of claims 9 to 11, characterized in that The first model includes n computing units connected in series and n+1 output terminals, the output of each computing unit in the n computing units connected in series serves as the input of the next computing unit, the input of the first computing unit in the n computing units is the first image data, and one output terminal in the n+1 output terminals outputs the second image data, where n is a positive integer.

13. The device according to claim 12, wherein The first model determines an output terminal for outputting the second image data through n+1 selection parameters, and the n+1 selection parameters correspond one-to-one to the n+1 output terminals.

14. The device according to claim 13, wherein The value of each of the n+1 selection parameters is 0 or 1, the sum of the n+1 selection parameters is 1, and the value of the selection parameter corresponding to the output terminal that outputs the second image data is 1.

15. An image processing device, characterized in that: The image processing apparatus comprises an input / output interface, a processor and a memory, wherein the processor is used to control the input / output interface to send and receive signals or information, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the image processing apparatus performs the image processing method according to any one of claims 1 to 7.

16. An image processing system, characterized in that: include: An image sensor for acquiring raw image data; An image signal processor ISP, configured to process the original image data according to a target ISP parameter group to obtain first image data, where the first image data is used for a first service; N image processing devices, configured to obtain N different second image data based on the first image data, wherein the data size of each of the N different second image data is greater than or equal to the data size of the first image data and less than or equal to the data size of the original image data; Furthermore, N different third image data are obtained according to the N different second image data, and the N different third image data are used for N different second services, where N is a positive integer greater than or equal to 1.

17. A smart device, characterized in that: The image processing device comprises the image processing device according to any one of claims 8 to 15.

Images