Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the image features of an image block, and enabling the image block to be obtained through the segmentation of a to-be-processed image; determining a processing main body for processing the image block according to the image features; if it is determined that the processing main body is an image signal processing unit, processing the image block through the image signal processing unit; and if it is determined that the processing subject is a deep learning accelerator, processing the image block through the deep learning accelerator. The scheme has relatively high applicability to the end side equipment with effective hardware resources.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0002] The growing demand for image processing has driven the development of artificial intelligence (AI) image processing, which can perform high-quality image enhancement and higher-level image processing tasks. However, end-side hardware struggles to support AI image processing, making it difficult to implement AI image processing in real time on hardware platforms with limited computing and storage resources.

[0003] At present, image signal processing (ISP) is combined with deep learning accelerator (DLA) in image processing, and a deep learning image enhancement network is used to replace specific functional modules in ISP to process the entire frame image.

[0004] However, although replacing specific functional modules in ISP with a deep learning image enhancement network can improve image processing quality, processing the entire frame of image with a deep learning image enhancement network requires high hardware resources and cannot be adapted to end-side devices with limited hardware resources. Summary of the Invention

[0005] In view of this, embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present application, an image processing method is provided, including: obtaining image features of an image block, wherein the image block is obtained by cutting out an image to be processed; determining a processing subject for processing the image block based on the image features; if it is determined that the processing subject is an image signal processing (ISP) unit, processing the image block through the ISP unit; if it is determined that the processing subject is a deep learning accelerator (DLA), processing the image block through the DLA.

[0007] According to a second aspect of an embodiment of the present application, an image processing device is provided, comprising: a branch control module, an image signal processing unit, and a deep learning accelerator; the branch control module is configured to obtain image features of an image block and determine, based on the image features, a processing subject for processing the image block, the image block being obtained by segmenting from an image to be processed; the image signal processing unit is configured to process the image block after being determined as the processing subject; and the deep learning accelerator is configured to process the image block after being determined as the processing subject.

[0008] According to the third aspect of the embodiment of the present application, an electronic device is provided, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation corresponding to the method described in the first aspect above.

[0009] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect above is implemented.

[0010] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to execute the method described in the first aspect above.

[0011] It can be seen from the above technical solution that image features can indicate the complexity of image blocks, and then based on the image features, the ISP unit or DLA can be determined as the processing subject for processing image blocks, so that image blocks with lower complexity can be processed by the ISP unit, while ensuring the image processing quality, reducing the image processing delay and reducing the requirements for hardware resources. Image blocks with higher complexity can be processed by the DLA to ensure the image processing quality. Therefore, while ensuring the image processing quality, the image processing delay and the requirements for hardware resources can be reduced, thereby adapting to end-side devices with efficient hardware resources and improving the applicability of the image processing method. In addition, the processing subject for processing image blocks can be determined based on image features. The image features can be obtained by statistical methods. Compared with the method of processing image blocks through neural networks to determine the processing subject, it can reduce calculations and access, and can also reduce the image processing delay and the requirements for hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 is a schematic diagram of an exemplary system used in one embodiment of the present application;

[0014] Figure 2 is a flowchart of an image processing method according to an embodiment of the present application;

[0015] Figure 3 is a flow chart of a network training method according to an embodiment of the present application;

[0016] Figure 4 is a schematic diagram of a network training process according to an embodiment of the present application;

[0017] Figure 5 This is a block diagram of the internal structure of an image processing device according to an embodiment of the present application;

[0018] Figure 6 is a block diagram of the internal structure of an image processing device according to another embodiment of the present application;

[0019] Figure 7 It is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The present application is described below based on examples, but the present application is not limited to these examples. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without describing these details. To avoid obscuring the essence of the present application, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0021] First, some nouns or terms that appear in the process of describing the embodiments of the present application are subject to the following explanations.

[0022] Image Signal Processing Unit: The Image Signal Processing (ISP) unit is a dedicated hardware for image processing. It converts the raw image from the image sensor through a series of fixed-function image processing modules for application in display and machine vision, etc.

[0023] Deep Learning Accelerator: A Deep Learning Accelerator (DLA) is a hardware device specifically designed to accelerate deep learning computations, such as a Graphics Processing Unit (GPU) or a Tensor Processing Unit (TPU). DLA can process voice data, text data, image data, and more through a neural network model. In this embodiment, DLA is used for image processing.

[0024] Shared Memory: Shared memory refers to a large amount of memory that can be accessed by different processors in a multi-processor computer system. In the embodiment of the present application, shared memory refers to a large amount of memory that can be accessed by both the ISP unit and the DLA.

[0025] Application environment of this application

[0026] The embodiment of the present application proposes an image processing solution based on ISP and DLA. The entire image processing solution is relatively universal and can be used in various hardware devices including ISP and DLA, such as mobile phones, tablet computers, personal computers, cameras, robots and cars. The image processing solution can perform various suitable types of processing on images, such as image noise reduction, image enhancement, image stabilization, etc. The image processing solution can process the original image from the image sensor, and can also process the image received through the network or uploaded locally. For illustrative purposes, the following description will mainly use electronic devices including image sensors as application scenarios. Those skilled in the art should understand that the embodiment of the present application can also be applied to other application scenarios.

[0027] Exemplary Systems

[0028] Figure 1 An exemplary system for the image processing method of the embodiment of the present application is shown. Figure 1 As shown, the system may include at least one image sensor 101, an image processing device 102 and an image application module 103. Figure 1 The example shown is a plurality of image sensors 101 .

[0029] The image sensor 101 may be a camera, a laser radar, etc. for image acquisition. After acquiring the original image, the image sensor 101 may send the original image to the image processing device 102 via a communication bus or a dedicated image transmission line.

[0030] After receiving the original image from the image sensor 101, the image processing device 102 may perform image noise reduction, image enhancement, image stabilization, and other processing on the original image to obtain a processed image, and then send the processed image to the image application module 103 via a communication bus or a dedicated image transmission line. As an optional example, in some embodiments, after receiving the original image, the image processing device 102 may process the original image based on the ISP and DLA by dividing the original image into multiple image blocks, performing complexity analysis on the image blocks, processing image blocks with lower complexity through the ISP, and processing image blocks with higher complexity through the DLA. After processing the multiple divided image blocks separately, the processed image blocks are spliced ​​together to obtain a processed image corresponding to the original image, and then the processed image is sent to the image application module 103.

[0031] After receiving the processed image, the image application module 103 can send the processed image to the display module for display. It can also further process the processed image based on machine vision to perform target recognition, object classification, etc. In some embodiments, the image application module 103 can perform lane recognition based on the processed image and then perform path planning based on the lane recognition results to achieve autonomous driving or driving assistance. In other embodiments, the image application module 103 can perform text recognition based on the processed image and then convert the text image in the processed image into editable text data.

[0032] The embodiment of the present application mainly focuses on the process of image processing performed by the image processing device 12, and the image processing method will be described in detail later.

[0033] Image processing methods

[0034] Based on the above system, an embodiment of the present application provides an image processing method, which can be executed by the image processing device 102. The image processing method is described in detail below through multiple embodiments.

[0035] Figure 2 This is a flow chart of an image processing method according to an embodiment of the present application. Figure 2 As shown, the image processing method includes the following steps:

[0036] Step 201: Acquire image features of an image block, where the image block is obtained by segmenting the image to be processed.

[0037] Image features can indicate the complexity of an image block, and the image features are obtained by performing statistics on pixels included in the image block.

[0038] By segmenting the image to be processed, we can obtain image blocks. The row pixels or column pixels of the image block can be equal to 1, that is, the image block can be a row of pixels or a column of pixels in the image to be processed. The image to be processed can be the original image captured by the image sensor or an image processed by the ISP unit or DLA.

[0039] The ISP unit includes multiple fixed-function image processing modules. Different image processing modules are used to perform different types of image processing. For example, an image processing module can perform white balance (WB), demosaicing (DMC), color matrix correction (CMC), and super-resolution (SR). The DLA can replace at least some of the image processing modules included in the ISP unit to process images. For example, the DLA can replace the ISP unit to perform demosaicing, super-resolution, and other processing on the image.

[0040] In one example, the ISP unit includes an image processing module 1, an image processing module 2, and an image processing module 3, which are arranged in sequence. The image processing module 1 is used to perform white balance processing on the image, the image processing module 2 is used to perform demosaicing processing on the image, and the image processing module 3 is used to perform super-resolution processing on the image. The DLA can replace the image processing module 2 to perform demosaicing processing on the image, and can also replace the image processing module 3 to perform super-resolution processing on the image. When white balance processing is performed on the image to be processed, the image to be processed is the original image captured by the image sensor. When super-resolution processing is performed on the image to be processed, the image to be processed is the processed image of the image processing module 2, or the processed image output after the DLA replaces the image processing module 2 to perform demosaicing processing.

[0041] Step 202: Determine a processing subject for processing the image block according to the image features.

[0042] ISP and DLA are suitable for processing image blocks of different complexities. ISP is suitable for processing image blocks with lower complexity, and has lower processing delay while ensuring image processing effects. DLA is suitable for processing image blocks with higher complexity, and can improve image processing effects compared to ISP, but the image processing delay is higher.

[0043] Because image features can indicate the complexity of an image block, the ISP and DLA can be analyzed based on the image features to determine which is more suitable for processing the image block. The ISP or DLA can then be determined as the processing entity for processing the image block. For example, when performing demosaicing or super-resolution on an image block, the ISP performs better than the DLA. If the ISP is determined to be the processing entity, step 203 is executed. If the DLA is determined to be the processing entity, step 204 is executed.

[0044] Step 203: If it is determined that the processing subject is the image signal processing unit, the image block is processed by the image signal processing unit.

[0045] If the ISP unit is determined to be the processing subject based on the image features of the image block, the image block is processed by the ISP unit.

[0046] Step 204: If it is determined that the processing subject is a deep learning accelerator, the image block is processed by the deep learning accelerator.

[0047] If the DLA is determined as the processing subject according to the image features of the image block, the image block is processed by the DLA.

[0048] It should be noted that the embodiment of the present application performs image processing in units of image blocks. The image to be processed can be divided into multiple image blocks. The processing subject of the image block is determined based on the image features of the image block. The processing subjects of different image blocks, that is, multiple image blocks cut out from the same image to be processed, part of the image blocks are processed by the ISP unit, and the other image blocks are processed by the DLA.

[0049] In an embodiment of the present application, image features can indicate the complexity of an image block, and then, based on the image features, an ISP unit or DLA can be determined as the processing subject for processing the image block, so that image blocks with lower complexity can be processed by the ISP unit, while ensuring image processing quality and reducing image processing delays and lowering the requirements for hardware resources. Image blocks with higher complexity can be processed by the DLA, ensuring image processing quality. Therefore, while ensuring image processing quality, image processing delays and requirements for hardware resources can be reduced, thereby adapting to end-side devices with efficient hardware resources and improving the applicability of the image processing method. In addition, the processing subject for processing the image block can be determined based on the image features. The image features can be obtained by statistical methods. Compared with the method of processing the image block through a neural network to determine the processing subject, it can reduce operations and access, and can also reduce image processing delays and requirements for hardware resources.

[0050] In one possible implementation, the image block may be stored in a shared memory through the ISP unit. After the DLA is determined as the processing subject of the image block, the DLA may read the image block from the shared memory and process the read image block.

[0051] The ISP unit is the leader of the image processing architecture, while the DLA assists the ISP unit, processing complex image blocks. After acquiring an image block, the ISP unit stores it in shared memory, which both the ISP unit and the DLA can access. When the DLA is designated as the processing agent, it can read and process the image block from the shared memory.

[0052] After the ISP unit stores the image block in the shared memory, if the ISP unit is determined as the processing subject, the ISP unit can read the image block from the shared memory and process the read image block. Alternatively, the ISP unit can cache the image block locally while storing the image block in the shared memory. After the ISP unit is determined as the processing subject, the ISP unit directly processes the locally cached image block.

[0053] After the ISP unit or DLA processes an image block, it can store the processed image block in shared memory. Specifically, after the ISP unit, as the processing entity, processes the image block, it stores the processed image block in shared memory. After the DLA, as the processing entity, processes the image block, it stores the processed image block in shared memory. Because multiple types of processing can be performed sequentially on an image block, and after completing the current type of processing, other types of processing can be performed on the processed image block. These other types of processing can be performed by the ISP unit or DLA, storing the processed image block in shared memory can reduce access to main memory, thereby improving image processing performance.

[0054] In an embodiment of the present application, since the image blocks can be processed by the ISP unit or the DLA, the image blocks are stored in the shared memory of the ISP unit and the DLA. The ISP unit and the DLA can read the image blocks from the shared memory and can also store the processed image blocks in the shared memory, thereby reducing the access to the main memory during the image processing process. The access to the shared memory has a lower latency than the access to the main memory, so the efficiency of image processing can be improved.

[0055] In one possible implementation, when obtaining the image features of an image block, the image block can be preprocessed by the ISP unit to obtain the image features. In the process of the ISP unit storing the image block in a shared memory, the pixels included in the image block can be counted to obtain the image features of the image block.

[0056] When processing the image to be processed, the image block will first enter the ISP unit, and the ISP unit will pre-process the pixels included in the image block, such as performing white balancing and demosaicing on the pixels included in the image block in turn. Based on the pre-processing results, the image features can be counted.

[0057] For an ISP with a higher degree of integration, image features can be counted at the interface between the ISP unit and the shared memory. That is, in the process of the ISP unit storing the image block in the shared memory, the pixels included in the image block are counted at the interface between the ISP unit and the shared memory to obtain the image features of the image block.

[0058] Image features are counted during the preprocessing of the image block by the ISP unit, or image features are counted at the interface between the ISP unit and the shared memory, which can be flexibly selected according to the integration level of the ISP.

[0059] In the embodiment of the present application, since the ISP unit pre-processes the image block, counts the image features of the image block during the pre-processing process, and then reads the image features from the ISP unit, there is no need to spend additional time to count the image features, thereby improving image processing efficiency. For a more integrated ISP, when the ISP unit stores the image block to the shared memory, the image features are counted at the interface between the ISP and the shared memory, eliminating the need to spend additional time to count the image features, thereby improving image processing efficiency.

[0060] In a possible implementation, the image features of the image block may include one or more of a histogram statistical result, a total difference sum, a variance, and a mean value of pixels in the image block.

[0061] In an embodiment of the present application, the histogram statistics, total differential sum, variance, and mean of the pixels in the image block can reflect the complexity of the image block as a whole. Using the histogram statistics, total differential sum, variance, and mean of the pixels in the image block as the image features of the image block can make the image features accurately reflect the complexity of the image block, and then determine the image block to be processed by the ISP unit or DLA based on the image features. While ensuring the processing quality of the image block, the efficiency of the entire frame image processing is improved and the hardware resources required for image processing are reduced. In addition, using the histogram statistics, total differential sum, variance, and mean as the image features of the image block reduces the amount of image feature data, reduces the amount of computation when determining the processing subject based on the image features, reduces the demand for hardware resources, and can improve the efficiency of image processing.

[0062] In one possible implementation, when determining the processing subject based on the image features of the image block, the image features of the image block can be input into a classification network to obtain a classification result output by the classification network, and then the processing subject for processing the image block is determined based on the classification result.

[0063] The classification network is a pre-trained neural network, such as a convolutional neural network (CNN). The classification network is used to predict, based on image features, whether an image block is appropriately processed by the ISP unit or the DLA. The classification results output by the classification network can be probabilities, for example, the classification results can include the probability that the image block is appropriately processed by the ISP unit and the probability that the image block is appropriately processed by the DLA.

[0064] In an embodiment of the present application, since the data volume of the image features is small, the image features are input into the classification network, and the complexity of the image block is analyzed by the classification network to determine whether the image block is processed by the ISP unit or the DLA. The processing subject can be determined with less time and hardware resources. While ensuring the efficiency of image processing, the accuracy of the classification results can be guaranteed, and then the image block is processed by an appropriate processing subject to ensure the quality of image block processing.

[0065] In one possible implementation, the classification result output by the classification network includes a first probability and at least one second probability. The first probability is the probability that the image block is appropriately processed by the ISP unit, and the second probability is the probability that the image block is appropriately processed by the DLA. Because the DLA can process image blocks using different image processing networks, and different image processing networks require different hardware resources to process image blocks, the classification result may include multiple second probabilities, with different second probabilities corresponding to the probabilities that the image block is appropriately processed by the DLA using different image processing networks.

[0066] If the first probability is greater than or equal to any second probability, it indicates that it is more appropriate to process the image block through the ISP unit than through the DLA, and the complexity of the image block is low in this case, and the ISP unit is determined as the processing entity. If any second probability among at least one second probability is greater than the first probability, it indicates that it is more appropriate to process the image block through the DLA than through the ISP unit, and the complexity of the image block is high in this case, and the DLA is determined as the processing entity.

[0067] In an embodiment of the present application, the classification result output by the classification network includes a first probability and at least one second probability. The first probability indicates the probability that the image block is appropriately processed by the ISP unit, and the second probability indicates the probability that the image block is appropriately processed by the DLA. When the first probability is greater than or equal to any second probability, the ISP unit is determined as the processing subject. When any second probability is greater than the first probability, the DLA is determined as the processing subject, ensuring that the ISP unit or DLA can be assigned to process the image block according to the complexity of the image block, while reducing the hardware resources required for processing the entire frame image, ensuring the image block processing quality and improving the efficiency of the entire frame image processing.

[0068] In one possible implementation, the classification result output by the classification network includes a first probability and multiple second probabilities. When at least one second probability among the multiple second probabilities is greater than the first probability, a target probability is determined from the multiple second probabilities so that the target probability is greater than or equal to any second probability among the second probabilities, and then the DLA processes the image block through the processing network corresponding to the target probability.

[0069] DLA can process image blocks using different processing networks. Different processing networks have different numbers of network channels and / or repeated operators, and thus require different hardware resources to process image blocks. However, the image processing quality of image blocks processed by different processing networks may be the same. Different processing networks correspond to different second probabilities, and the suitability of processing an image block by a processing network is positively correlated with the second probability corresponding to the processing network.

[0070] In one example, the classification result includes a first probability P1, a second probability P2, and a second probability P3, wherein the second probability P2 corresponds to processing network 1, and the second probability P3 corresponds to processing network 2. If P1≥P2 and P1≥P3, the ISP unit is determined as the processing subject, and the image block is processed by the ISP unit. If at least one of P2 and P3 is greater than P1, the DLA is determined as the processing subject, and the image block is processed by the DLA. For example, if P2>P1>P3, the DLA processes the image block through the processing network 1 corresponding to the second probability P2, and if P3>P2=P1, the DLA processes the image block through the processing network 1 corresponding to the second probability P3.

[0071] A processing network may include multiple repeating operators that perform the same processing on an image block. Different processing networks may include different numbers of repeating operators. For example, processing network 1 includes one first operator, two second operators, and one third operator, while processing network 2 includes one first operator, four second operators, and one third operator. A processing network may include multiple network channels. Different processing networks may include different numbers of network channels. For example, processing network 1 includes 16 network channels, while processing network 2 includes 32 network channels. The hardware resources required for image block processing are positively correlated with the number of network channels and repeating operators in the processing network. For example, processing an image block through processing network 2 requires more hardware resources than processing an image block through processing network 1. However, the image processing quality of an image block is not positively correlated with the number of network channels and repeating operators in the processing network. For example, processing network 1 and processing network 2 provide the same image processing quality for the same image block.

[0072] It should be noted that the processing network can be any suitable type of neural network, for example, the processing network can be a convolutional neural network. The embodiments of the present application do not limit the network type of the processing network.

[0073] In an embodiment of the present application, DLA can process image blocks through different processing networks. Different processing networks have different numbers of network channels and / or repeated operators, so different processing networks require different hardware resources to process image blocks, but different processing networks may have the same image processing quality for the same image block. Therefore, the classification network can analyze the image processing quality of different processing networks for image blocks based on image features, and make the second probability corresponding to the processing network that requires fewer hardware resources larger under the premise of the same image processing quality, thereby reducing the hardware resources required for image processing while ensuring the image processing quality.

[0074] In one possible implementation, the classification result output by the classification network includes a first probability and three second probabilities, where the first second probability corresponds to a small processing network, the second second probability corresponds to a medium processing network, and the third second probability corresponds to a large processing network. The weight of the small processing network is a subset of the weight of the medium processing network, and the weight of the medium processing network is a subset of the weight of the large processing network.

[0075] The number of network channels of the small processing network is smaller than that of the medium processing network, and / or the number of repetitive operators of the small processing network is smaller than that of the medium processing network. The number of network channels of the medium processing network is smaller than that of the large processing network, and / or the number of repetitive operators of the medium processing network is smaller than that of the large processing network.

[0076] In one example, the classification result includes a first probability P1, a second probability P2, a second probability P3, and a second probability P4. The second probability P2 corresponds to a small processing network, the second probability P3 corresponds to a medium processing network, and the second probability P4 corresponds to a large processing network. When determining that the DLA is the processing subject, if P2>P3 and P2>P4, the image block is processed by the small processing network; if P3>P2 and P3>P4, the image block is processed by the medium processing network; if P4>P2 and P4>P3, the image block is processed by the large processing network.

[0077] The small processing network, the medium processing network and the large processing network have the same structure and operators. The weight of the small processing network is a subset of the weight of the medium processing network, so the small processing network reduces the number of network channels and / or repeated operators based on the medium processing network. The weight of the medium processing network is a subset of the weight of the large processing network, so the medium processing network reduces the number of network channels and / or repeated operators based on the large processing network.

[0078] In an embodiment of the present application, the weights of the small processing network are a subset of the weights of the medium processing network, and the weights of the medium processing network are a subset of the weights of the large processing network. After the DLA is determined as the processing subject, the DLA loads the large processing network. Based on the target probability determined from each second probability, if the target probability corresponds to the large processing network, the image block is processed by the large processing network. If the target probability corresponds to the medium processing network, the portion of the large processing network other than the medium processing network is disabled, thereby processing the image block by the medium processing network. If the target probability corresponds to the small processing network, the portion of the large processing network other than the small processing network is disabled, thereby processing the image block by the small processing network. It can be seen that when the DLA switches the processing network, there is no need to dynamically switch weights. It only needs to load the large processing network and switch the small processing network, medium processing network, and large processing network to process the image block by modifying the parameters of the DLA, thereby reducing a large amount of weights and the overhead of switching weights.

[0079] In one possible implementation, the small processing network, the medium processing network, and the large processing network can be trained by dynamic optimization, so that the weight of the small processing network is a subset of the weight of the medium processing network, and the weight of the medium processing network is a subset of the weight of the large processing network.

[0080] Figure 3 This is a flow chart of a network training method according to an embodiment of the present application, which is used to train a small processing network, a medium processing network, and a large processing network. Figure 3 As shown, the network training method includes the following steps:

[0081] Step 301: Train the performance target network using the training data set.

[0082] The training data set includes multiple training image blocks for network training. For example, the training data set includes 10,000 training image blocks.

[0083] The performance target network can perform the same type of image processing as the small processing network, the medium processing network, and the large processing network. When training the performance target network, the performance target network can be obtained by training the initialization network using the training dataset, or by training the pre-trained network using the training dataset.

[0084] After obtaining the performance target network, the training image blocks included in the training dataset are processed using the performance target network to determine the image quality of the training image blocks in the training dataset processed by the performance target network. The training image blocks included in the training dataset are processed using the ISP unit to determine the image quality of the training image blocks in the training dataset processed by the ISP unit. This allows for determining which training image blocks in the training dataset were better processed by the ISP unit and which were better processed by the performance target network.

[0085] Figure 4 FIG. 1 is a schematic diagram of a network training process according to an embodiment of the present application. Figure 4 As shown in the figure, the performance target network is obtained by training the training dataset. The training image blocks in the training dataset that are better processed by the performance target network constitute the DLA dataset, and the training image blocks in the training dataset that are better processed by the ISP unit constitute the ISP dataset. The intersection of the DLA dataset and the ISP dataset is empty, and the union of the DLA dataset and the ISP dataset is the training dataset. For example, the training dataset includes 10,000 training image blocks, the DLA dataset includes 9,000 training image blocks, and the ISP dataset includes 1,000 training image blocks.

[0086] Step 302: After reducing the number of network channels and / or repeated operators of the performance target network, training is performed using a training data set to obtain a first network.

[0087] After obtaining the performance target network, without changing the operators and the order of operators, the number of network channels and / or repeated operators of the performance target network is reduced and retrained using the training data set to obtain the first network.

[0088] After obtaining the first network, the training image blocks included in the DLA dataset can be processed by the first network to determine the image quality after the first network processes the training image blocks in the DLA dataset. Figure 4As shown, the training image blocks in the DLA dataset that are better processed by the first network constitute the third DLA data subset, and the training image blocks in the DLA dataset that are better processed by the performance target network constitute the fourth DLA data subset.

[0089] Step 303: Train the first network using the DLA dataset to obtain a small processing network.

[0090] The DLA dataset is a subset of the training dataset, and the image quality of the training image blocks included in the DLA dataset after being processed by the performance target network is better than the image quality after being processed by the ISP unit.

[0091] After obtaining the small processing network, the training image blocks included in the DLA dataset can be processed by the small processing network to determine the image quality after the training image blocks in the DLA dataset are processed by the small processing network. Figure 4 As shown, the fifth DLA data subset consists of training image blocks in the DLA dataset that are processed better by the small processing network, while the first DLA data subset consists of training image blocks in the DLA dataset that are processed better by the performance target network. The small processing network is obtained by fine-tuning the first network using the DLA dataset. Compared to the first network, the small processing network can process more training image blocks in the training dataset with higher quality. That is, the fifth DLA data subset includes more training image blocks than the third DLA data subset.

[0092] Step 304: After increasing the number of network channels and / or repeated operators of the small processing network and fixing the weights of the small processing network, training is performed using the DLA dataset to obtain a second network.

[0093] After obtaining the small processing network, the number of network channels and / or repeated operators in the small processing network is increased without changing the operators or the order of operators. The weights of the small processing network are fixed and then trained using the DLA dataset to obtain a second network. The second network is equivalent to the performance target network with the number of network channels and / or repeated operators reduced without changing the operators or the order of operators, and the second network includes the small processing network.

[0094] After obtaining the second network, the training image blocks included in the DLA dataset can be processed by the second network to determine the image quality after the second network processes the training image blocks in the DLA dataset. Figure 4As shown, the training image blocks in the DLA dataset that are better processed by the second network constitute the sixth DLA data subset, the training image blocks in the DLA dataset that are better processed by the performance target network constitute the seventh DLA data subset, the sixth DLA data subset includes the fifth DLA data subset, and the sixth DLA data subset includes more training image blocks than the fifth DLA data subset.

[0095] The number of network channels and / or repeated operators of the small processing network is increased, and fine-tuning is performed using the DLA data set to obtain a second network. The second network has an increased network scale relative to the small processing network, and can better process more training image blocks in the training data set compared to the small processing network. That is, the sixth DLA data subset includes more training image blocks than the fifth DLA data subset.

[0096] Step 305: After fixing the weights of the small-sized processing network in the second network, the second network is trained using the first DLA data subset to obtain a medium-sized processing network.

[0097] The first DLA data subset is a subset of the DLA data set, and the image quality of the training image blocks included in the first DLA data subset after being processed by the performance target network is better than the image quality after being processed by the small processing network.

[0098] like Figure 4 As shown, for the training image blocks in the fifth DLA data subset, the image quality processed by the small processing network is better than the image quality processed by the performance target network. Therefore, the union of the first DLA data subset and the fifth DLA data subset is the DLA data set, and the intersection of the first DLA data subset and the fifth DLA data subset is empty. For example, if the DLA data set includes 9,000 training image blocks and the fifth DLA data subset includes 2,000 training image blocks, then the first DLA data subset includes 7,000 training image blocks.

[0099] After obtaining the medium-sized processing network, the training image blocks included in the DLA dataset can be processed by the medium-sized processing network to determine the image quality after the medium-sized processing network processes the training image blocks in the DLA dataset. Figure 4 As shown, the training image blocks in the DLA dataset that are better processed by the medium processing network constitute the eighth DLA data subset, and the training image blocks in the DLA dataset that are better processed by the performance target network constitute the second DLA data subset. The medium processing network is obtained by fine-tuning the second network using the first DLA data subset. Compared with the second network, the medium processing network can process more training image blocks in the training dataset with better quality, that is, the eighth DLA data subset includes the sixth DLA data subset, and the eighth DLA data subset includes more training image blocks relative to the sixth DLA data subset.

[0100] Step 306: Increase the number of network channels and / or repeated operators of the medium-sized processing network, fix the weights of the medium-sized processing network, and then perform training using the second DLA data subset to obtain a large-sized processing network.

[0101] After obtaining the medium-sized processing network, the number of network channels and / or repeated operators in the medium-sized processing network is increased without changing the operators or the order of the operators. The weights of the medium-sized processing network are fixed and then trained on the second DLA data subset to obtain a large-sized processing network. The large-sized processing network has the same number of network channels as the performance target network, and the large-sized processing network has the same number of repeated operators as the performance target network.

[0102] The second DLA data subset is a subset of the first DLA data subset, and the image quality of the training image blocks included in the second DLA data subset after being processed by the performance target network is better than the image quality after being processed by the medium processing network. Figure 4 As shown, the union of the second DLA data subset and the eighth DLA data subset is the DLA dataset. For example, the DLA dataset includes 9,000 training image blocks, the eighth DLA data subset includes 7,000 training image blocks, and the second DLA data subset includes 2,000 training image blocks.

[0103] The image quality of the image blocks in the second DLA data subset processed by the medium-sized processing network is poor. The large-scale processing network is obtained by fine-tuning the medium-sized processing network through the second DLA data subset, so that the large-scale processing network has better image quality for the training image blocks in the second DLA data subset, and thus the large-scale processing network has better image quality for the training image blocks in the DLA data set. Therefore, the DLA data set includes more training image blocks relative to the eighth DLA data subset.

[0104] like Figure 4 As shown, the image quality of the training image blocks in the DLA dataset after being processed by the large processing network is better than that after being processed by the ISP unit, and the image quality of the training image blocks in the ISP dataset after being processed by the ISP unit is better than that after being processed by the large processing network.

[0105] In an embodiment of the present application, a small processing network, a medium processing network, and a large processing network are trained through dynamic optimization, such that the weights of the small processing network are a subset of the weights of the medium processing network, and the weights of the medium processing network are a subset of the weights of the large processing network. Compared to the performance target network, the small processing network has superior image processing quality for training image blocks in the fifth DLA data subset, the medium processing network has superior image processing quality for training image blocks in the eighth DLA data subset, and the large processing network has superior image processing quality for training image blocks in the DLA dataset. The small and medium processing networks reduce the amount of computation required compared to the performance target network. The weights of the small processing network are a subset of the weights of the medium processing network, and the weights of the medium processing network are a subset of the weights of the large processing network. When switching networks using the DLA, there is no need to dynamically swap weights. After loading the large processing network, the small and medium processing networks can be used by modifying DLA parameters. This reduces the weights and weight switching overhead while maintaining hardware utilization.

[0106] Image processing device

[0107] Figure 5 FIG. 1 is a block diagram of the internal structure of the image processing device 102 according to an embodiment of the present application. Figure 5 As shown, the image processing apparatus 102 includes: a branch control module 501 , an ISP unit 502 and a DLA 503 .

[0108] The branch control module 501 can obtain image features of an image block and, based on the image features, determine a processing entity for processing the image block. The image block is obtained by segmenting the image to be processed. After the branch control module 501 determines the ISP unit 502 as the processing entity, the ISP unit 502 processes the image block. After the branch control module 501 determines the DLA 503 as the processing entity, the DLA 503 processes the image block.

[0109] In an embodiment of the present application, image features can indicate the complexity of an image block. The branch control module 501 can determine the ISP unit or DLA as the processing subject for processing the image block based on the image features, so that image blocks with lower complexity can be processed by the ISP unit, while ensuring the image processing quality and reducing the image processing delay and reducing the requirements for hardware resources. Image blocks with higher complexity can be processed by the DLA to ensure the image processing quality. Therefore, while ensuring the image processing quality, the image processing delay and the requirements for hardware resources can be reduced, thereby adapting to end-side devices with efficient hardware resources and improving the applicability of the image processing method. In addition, the processing subject for processing the image block is determined based on the image features. The image features can be obtained by statistical methods. Compared with the method of processing the image block through a neural network to determine the processing subject, it can reduce calculations and accesses, and can also reduce the image processing delay and the requirements for hardware resources.

[0110] In a possible implementation, the image processing apparatus 102 may further include a shared memory of the ISP unit 502 and the DLA 503 , so that the ISP unit 502 and the DLA 503 exchange data through the shared memory.

[0111] Figure 6 FIG. 1 is a block diagram of the internal structure of an image processing device 102 according to another embodiment of the present invention. Figure 6 As shown, the image processing apparatus 102 includes a shared memory 504. The ISP unit 502 can store image blocks in the shared memory 504. When the branch control module 501 determines the DLA 503 as the processing subject, the DLA 503 can read the image blocks from the shared memory 504 and process the read image blocks. When the branch control module 501 determines the ISP unit 502 as the processing subject, the ISP unit 502 can read the image blocks from the shared memory 504 and process the read image blocks. Alternatively, the ISP unit 502 can directly process the locally cached image blocks.

[0112] In an embodiment of the present application, since the image block can be processed by the ISP unit 502 or the DLA503, the image block is stored in the shared memory 504 of the ISP unit 502 and the DLA503. The ISP unit 502 and the DLA503 can read the image block from the shared memory 504, and can also store the processed image block in the shared memory 504, thereby reducing the access to the main memory during the image processing process. The access to the shared memory has a lower latency than the access to the main memory, so the efficiency of image processing can be improved.

[0113] In one possible implementation, Figure 6 As shown, the branch control module 501 may include a classification unit 5011 and a branch unit 5012 .

[0114] The ISP unit 502 may perform pre-processing such as white balancing and demosaicing on the image block. During the pre-processing process, image features of the image block may be counted, and then the classification unit 5011 may obtain the image features from the ISP unit 502. Alternatively, during the process of the ISP unit 502 storing the image block in the shared memory 504, the image features of the image block may be counted at the interface between the ISP unit 502 and the shared memory 504, and then the classification unit 5011 may obtain the image features of the image block from the interface.

[0115] After acquiring the image features, the classification unit 5011 may input the image features into a classification network to obtain a classification result, and send the classification result to the branch unit 5012 .

[0116] After receiving the classification result, the branch unit 5012 determines the better branch for processing the image block, that is, whether to process the image block through the ISP unit 502 or through the DLA503. If processing the image block through DLA503 is better, it further determines which level of dynamic network DLA503 uses to process the image block, such as determining the processing network used to process the image block from a small processing network, a medium processing network, and a large processing network.

[0117] In one example, Figure 6 As shown, after acquiring the image block, the ISP unit 502 pre-processes the image block through one or more image processing modules and stores the pre-processed image block in the shared memory 504. The classification unit 5011 obtains the image features counted during the pre-processing process from the ISP unit 502. The classification unit 5011 determines the processing subject based on the image features.

[0118] If the ISP unit 502 is determined to be the processing subject, the branch unit 5012 reads the image block from the shared memory 504 and sends the read image block to the ISP unit 502. The ISP unit 502 processes the received image block and stores the processed image block in the shared memory 504.

[0119] If DLA 503 is determined to be the processing entity, branch unit 5012 reads the image block from shared memory 504 and sends the read image block to DLA 503. Based on the classification result, DLA 503 processes the received image block through the corresponding processing network and stores the processed image block in shared memory 504. In one example, based on the classification result, DLA 503 processes the image block through a small processing network S, a medium processing network M, or a large processing network F. The weights of the small processing network S are a subset of the weights of the medium processing network M, and the weights of the medium processing network M are a subset of the weights of the large processing network F. DLA 503 only needs to load the large processing network. By blocking some weights in the large processing network F, the image block can be processed through the small processing network S or the medium processing network M, thereby reducing the number of weights and the overhead of switching weights.

[0120] It should be noted that the image processing device 102 of this embodiment is used to implement the image processing method in the aforementioned method embodiment. The interaction between the various modules and units in the image processing device 102 can be referred to the description in the aforementioned image processing method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0121] electronic devices

[0122] Figure 7 This is a schematic block diagram of an electronic device provided in an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device. Figure 7 As shown, the electronic device may include: a processor 702, a communications interface 704, a memory 706, and a communication bus 708.

[0123] The processor 702 , the communication interface 704 , and the memory 706 communicate with each other via a communication bus 708 .

[0124] The communication interface 704 is used to communicate with other electronic devices or servers.

[0125] The processor 702 is configured to execute the program 710 , and specifically may execute the relevant steps in any of the aforementioned image processing method embodiments.

[0126] Specifically, the program 710 may include program codes, which include computer operation instructions.

[0127] The processor 702 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0128] RISC-V is an open-source instruction set architecture based on the principles of the Reduced Instruction Set (RISC). It can be applied to various fields, including microcontrollers and FPGA chips. Specifically, it has applications in areas such as IoT security, industrial control, mobile phones, and personal computers. Designed with small size, high speed, and low power consumption in mind, it is particularly suitable for modern computing devices such as warehouse-scale cloud computers, high-end mobile phones, and tiny embedded systems. With the rise of the artificial intelligence (AI) Internet of Things (AIoT), the RISC-V instruction set architecture has also received increasing attention and support, and is expected to become the next generation of widely used CPU architecture.

[0129] The computer operation instructions in the embodiments of the present application may be computer operation instructions based on the RISC-V instruction set architecture. Correspondingly, the processor 702 may be designed based on the RISC-V instruction set. Specifically, the processor chip in the electronic device provided in the embodiments of the present application may be a chip designed using the RISC-V instruction set. The chip may execute executable code based on the configured instructions, thereby implementing the key generation method in the above embodiments.

[0130] The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0131] The program 710 may be specifically configured to enable the processor 702 to execute the image processing method in any of the aforementioned embodiments.

[0132] The specific implementation of each step in program 710 can be found in the corresponding descriptions of the corresponding steps and units in any of the aforementioned image processing method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that for the sake of convenience and brevity, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the aforementioned method embodiments, and will not be repeated here.

[0133] Through the electronic device of the embodiment of the present application, the image features can indicate the complexity of the image block, and then the ISP unit or DLA can be determined as the processing subject for processing the image block based on the image features, so that the image blocks with lower complexity can be processed by the ISP unit, while ensuring the image processing quality, reducing the image processing delay and reducing the requirements for hardware resources. The image blocks with higher complexity can be processed by the DLA to ensure the image processing quality. Therefore, while ensuring the image processing quality, the image processing delay and the requirements for hardware resources can be reduced, thereby adapting to the terminal side devices with efficient hardware resources and improving the applicability of the image processing method. In addition, the processing subject for processing the image block is determined based on the image features. The image features can be obtained by statistical methods. Compared with the method of processing the image block through a neural network to determine the processing subject, it can reduce calculations and accesses, and can also reduce the image processing delay and the requirements for hardware resources.

[0134] Computer storage media

[0135] This application also provides a computer-readable storage medium storing instructions for causing a machine to execute the image processing method described herein. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0136] In this case, the program code read from the storage medium itself can realize the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of this application.

[0137] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0138] Computer program product

[0139] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform any corresponding operation in the above-mentioned multiple method embodiments.

[0140] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0141] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or can be implemented as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0142] It should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0143] It should be understood that the various embodiments in this specification are described in a progressive manner. References to the same or similar parts of each embodiment can be made to the other embodiments. The embodiments focus on the differences from other embodiments. In particular, the method embodiments are generally similar to the methods described in the device and system embodiments, so their description is relatively simple. For relevant details, references to the descriptions of other embodiments can be made.

[0144] It should be understood that the foregoing description of this specification is based on specific embodiments. Other embodiments are within the scope of the claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0145] It should be understood that an element described herein in the singular or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown herein as separate may be combined into a single module or element, and modules or elements described or shown herein as single may be split into multiple modules or elements.

[0146] It should also be understood that the terms and expressions used herein are for descriptive purposes only, and the one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or portions thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.

Claims

1. An image processing method, comprising: Acquiring image features of an image block, where the image block is obtained by segmenting the image to be processed; determining a processing subject for processing the image block according to the image feature; If it is determined that the processing subject is an image signal processing (ISP) unit, processing the image block by the ISP unit; If it is determined that the processing subject is a deep learning accelerator, the image block is processed by the DLA.

2. The method according to claim 1, further comprising: Storing the image block in a shared memory through the ISP unit; After the DLA is determined as the processing subject, the DLA reads the image block from the shared memory.

3. The method according to claim 2, wherein: The obtaining of image features of the image block includes: Preprocessing the image block by the ISP unit to obtain the image features; or, When the ISP unit stores the image block in the shared memory, the pixels included in the image block are counted to obtain the image features.

4. The method according to claim 1, wherein The image feature includes at least one of a histogram statistical result, a total difference sum, a variance, and a mean of pixels in the image block.

5. The method according to any one of claims 1 to 4, wherein: The determining, based on the image feature, a processing subject for processing the image block includes: Inputting the image features into a classification network to obtain a classification result output by the classification network; A processing subject for processing the image block is determined according to the classification result.

6. The method according to claim 5, wherein: The classification result includes a first probability and at least one second probability; The determining, according to the classification result, a processing subject for processing the image block includes: If the first probability is greater than or equal to any of the second probabilities, determining that the processing subject is the ISP unit; If any of the second probabilities is greater than the first probability, the processing subject is determined to be the DLA.

7. The method according to claim 6, wherein: The classification result includes a plurality of second probabilities; The processing of the image block by the DLA includes: processing the image block by a processing network corresponding to a target probability among the multiple second probabilities, wherein the target probability is greater than or equal to any second probability among the multiple second probabilities, different second probabilities correspond to different processing networks, and different processing networks have different numbers of network channels and / or repeated operators.

8. The method according to claim 7, wherein: The classification result includes three second probabilities, the first second probability corresponds to a small processing network, the second second probability corresponds to a medium processing network, and the third second probability corresponds to a large processing network. The weight of the small processing network is a subset of the weight of the medium processing network, and the weight of the medium processing network is a subset of the weight of the large processing network.

9. The method according to claim 8, wherein The small processing network, the medium processing network, and the large processing network are trained in the following manner: Train the performance target network using the training dataset; After reducing the number of network channels and / or repeated operators of the performance target network, training is performed using the training data set to obtain a first network; Training the first network using a DLA dataset to obtain the small processing network, wherein the DLA dataset is a subset of the training dataset, and the image quality of the training image blocks included in the DLA dataset after being processed by the performance target network is better than the image quality after being processed by the ISP unit; Increasing the number of network channels and / or repeated operators of the small processing network and fixing the group weight of the small processing network, and then training the network using the DLA dataset to obtain a second network; After fixing the weights of the small processing network in the second network, training the second network using a first DLA data subset to obtain the medium processing network, wherein the first DLA data subset is a subset of the DLA data set, and the image quality of the training image blocks included in the first DLA data subset after being processed by the performance target network is better than the image quality after being processed by the small processing network; After increasing the number of network channels and / or repeated operators of the medium-sized processing network and fixing the weights of the medium-sized processing network, training is performed using a second DLA data subset to obtain the large-sized processing network, where the second DLA data subset is a subset of the first DLA data subset, and the image quality of the training image blocks included in the second DLA data subset after being processed by the performance target network is better than the image quality after being processed by the medium-sized processing network. The large-sized processing network has the same number of network channels and repeated operators as the performance target network.

10. An image processing device, comprising: Branch control module, image signal processing unit and deep learning accelerator; The branch control module is configured to obtain image features of an image block and determine a processing subject for processing the image block according to the image features, wherein the image block is obtained by segmenting the image to be processed; The image signal processing unit is configured to process the image block after being determined as the processing subject; The deep learning accelerator is used to process the image block after being determined as the processing subject.

11. The apparatus according to claim 10, further comprising: Shared memory; The image signal processing device is used to store the image block in the shared memory; The deep learning accelerator is configured to read the image block from the shared memory after being determined as the processing subject.

12. An electronic device comprising: Processor, memory, communication interface and communication bus, the processor, memory and communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to any one of the methods according to claims 1-9.

13. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

14. A computer program product comprising computer instructions, the computer instructions instructing a computing device to execute the method according to any one of claims 1 to 9.

Images