Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring image data; carrying out sparsity statistics on the image data based on the convolution size of the trained pulse neural network to obtain a sparsity statistics result; determining valid data from the image data based on a sparsity statistical result; inputting the effective data into a pulse neural network for calculation to obtain a plurality of output results; and fusing the plurality of output results to obtain a fused result.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to an image processing method, apparatus, electronic device, and storage medium. Background Art

[0002] Currently, neural networks are widely used in fields such as image processing, object detection, and pattern recognition. In the prior art, as a brain-inspired computing model, the Spiking Neural Network (SNN) model has the characteristics of asynchrony, sparsity, and low power consumption, and can complete efficient computing under low-power conditions, making it suitable for embedded visual perception and intelligent reasoning. However, the existing SNN computing architecture still relies on the memory-computation separation mode, and the energy consumption overhead caused by data transfer is still relatively high, which limits the popularization of SNN in practical applications. Summary of the Invention

[0003] In view of this, the present disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0004] One aspect of the present disclosure provides an image processing method, including: obtaining image data; performing sparsity statistics on the image data based on the convolution size of a trained spiking neural network to obtain a sparsity statistics result; determining valid data from the image data based on the sparsity statistics result; inputting the valid data into the spiking neural network for calculation to obtain a plurality of output results; and fusing the plurality of output results to obtain a fused result.

[0005] According to an embodiment of the present disclosure, the sparsity statistics result includes the positions and the number of valid data. Performing sparsity statistics on the image data based on the convolution size of a trained spiking neural network to obtain a sparsity statistics result includes: dividing the image data into a plurality of image blocks based on the convolution size of a trained spiking neural network, and extracting a plurality of data to be operated on from each image block; performing an OR operation on the plurality of data to be operated on to obtain a plurality of OR operation results; determining the data to be operated on as valid data when the OR operation result is not zero; and determining the positions and the number of valid data based on the plurality of OR operation results.

[0006] According to an embodiment of the present disclosure, dividing the image data into a plurality of image blocks based on the convolution size of a trained spiking neural network, and extracting a plurality of data to be operated on from each image block includes: adding a zero element to each of the start and end of each row of data in the image data to obtain a first matrix; adding a zero element to each of the start and end of each column of data in the first matrix to obtain a second matrix; and dividing the second matrix into a plurality of image blocks based on the convolution size of the spiking neural network, and extracting a plurality of data to be operated on from each image block.

[0007] According to an embodiment of the present disclosure, determining valid data from image data based on the sparsity statistical result includes: for each image block, when the number of valid data is not zero, determining the valid data according to the positions of the valid data.

[0008] According to an embodiment of the present disclosure, inputting the valid data into a spiking neural network for calculation to obtain multiple output results includes: storing the valid data in a first buffer; inputting the valid data in the first buffer into the spiking neural network for calculation to obtain output results; when new valid data is received while processing the valid data in the first buffer, storing the new valid data in a second buffer; when the processing of the valid data in the first buffer is completed, inputting the valid data in the second buffer into the spiking neural network for calculation to obtain output results.

[0009] According to an embodiment of the present disclosure, it further includes: when new valid data is received while processing the valid data in the second buffer, storing the new valid data in the first buffer; when the processing of the valid data in the second buffer is completed, inputting the valid data in the first buffer into the spiking neural network for calculation to obtain output results.

[0010] According to an embodiment of the present disclosure, the spiking neural network includes a first spiking neural network and a second spiking neural network. Inputting the valid data into the spiking neural network for calculation to obtain multiple output results includes: inputting the valid data into the first spiking neural network to obtain multiple data to be input, and the first spiking neural network includes multiple different spiking neural networks; inputting the multiple data to be input into the second spiking neural network to obtain multiple output results.

[0011] Another aspect of the present disclosure further provides an image processing apparatus, including: an acquisition module, configured to acquire image data; a statistics module, configured to perform sparsity statistics on the image data based on the convolution size of a trained spiking neural network to obtain a sparsity statistical result; a determination module, configured to determine valid data from the image data based on the sparsity statistical result; a calculation module, configured to input the valid data into the spiking neural network for calculation to obtain multiple output results; a fusion module, configured to fuse the multiple output results to obtain a fused result.

[0012] Another aspect of the present disclosure provides an electronic device, including: one or more processors; a storage device, configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the above method.

[0013] Another aspect of the present disclosure further provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor is caused to execute the above method.

[0014] According to an embodiment of the present disclosure, by using the sparsity statistical result, valid data is determined from the image data, and the valid data is input into the spiking neural network, reducing the redundant calculation caused by invalid data and improving the neural network calculation efficiency. Moreover, by operating alternately in two buffer spaces, high-efficiency parallel computing is achieved, which is applicable to low-power and real-time intelligent computing scenarios.

[0015] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Through the following description of the embodiments of the present disclosure with reference to the drawings, the above and other objects, features, and advantages of the present disclosure will become clearer. In the drawings:

[0017] Figure 1 Schematically shows an application scenario diagram of an image processing method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0018] Figure 2 Schematically shows a flowchart of an image processing method according to an embodiment of the present disclosure;

[0019] Figure 3 Schematically shows a flowchart of an image processing method according to another embodiment of the present disclosure;

[0020] Figure 4 Schematically shows a structural block diagram of a neural network model suitable for implementing an image processing method according to an embodiment of the present disclosure;

[0021] Figure 5 Schematically shows a structural block diagram of a neural network model suitable for implementing an image processing method according to another embodiment of the present disclosure;

[0022] Figure 6 Schematically shows a structural block diagram of a neural network model suitable for implementing an image processing method according to another embodiment of the present disclosure;

[0023] Figure 7 Schematically shows a structural block diagram of a neural network model suitable for implementing an image processing method according to another embodiment of the present disclosure;

[0024] Figure 8 Schematically shows a structural block diagram of an image processing system according to an embodiment of the present disclosure;

[0025] Figure 9 Schematically shows a structural block diagram of an image processing system according to another embodiment of the present disclosure;

[0026] Figure 10 Schematically shows a structural block diagram of an image processing system according to another embodiment of the present disclosure;

[0027] Figure 11 Schematically shows a structural block diagram of an image processing apparatus according to an embodiment of the present disclosure; and

[0028] Figure 12 Schematically shows a block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of the present disclosure. Detailed implementation manners

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, many specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it is obvious that one or more embodiments can be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present disclosure.

[0030] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] In cases where expressions such as "at least one of A, B, and C, etc." are used, generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0033] Embodiments of the present disclosure provide an image processing method, the method including: acquiring image data; performing sparsity statistics on the image data based on the convolution size of a trained spiking neural network to obtain a sparsity statistics result; determining valid data from the image data based on the sparsity statistics result; inputting the valid data into the spiking neural network for calculation to obtain a plurality of output results; and fusing the plurality of output results to obtain a fused result.

[0034] By using the sparsity statistical results to determine valid data from the image data and inputting the valid data into the spiking neural network, the redundant calculations caused by invalid data can be reduced, and the computing efficiency of the neural network can be improved.

[0035] Figure 1 FIG. schematically shows an application scenario diagram of an image processing method, apparatus, device, medium, and program product according to an embodiment of the present disclosure.

[0036] As Figure 1 shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0037] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0038] The terminal devices 101, 102, 103 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0039] The server 105 may be a server providing various services, such as a background management server (only as an example) that supports the websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process data such as received images, etc., and feedback the processing results (such as numbers or letters identified according to the image data, etc.) to the terminal devices.

[0040] It should be noted that the image processing method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the image processing apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 105. The image processing method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the image processing apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105.

[0041] It should be understood that Figure 1 the numbers of the terminal devices, networks, and servers in [[ ]] are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, networks, and servers.

[0042] Figure 2 A flowchart of an image processing method according to an embodiment of the present disclosure is schematically shown.

[0043] As Figure 2 shown, the image processing method of this embodiment includes operations S210 to S250.

[0044] In operation S210, image data is acquired.

[0045] For example, environmental image data can be collected by a pulsed image sensor.

[0046] Specifically, the pulsed image sensor can be a configurable parameter pulsed image sensor, and the exposure time, gain, and frame rate of the image output can be configured. The pulsed image sensor performs continuous imaging in a global exposure manner, and the resolution of the image can be set to 128×128, the exposure time can be 5 ms, and the gain can be 3. This parameter configuration method can better meet the imaging requirements under different lighting conditions.

[0047] In operation S220, based on the convolution size of the trained pulsed neural network, sparsity statistics are performed on the image data to obtain a sparsity statistics result.

[0048] The convolution size of the trained pulsed neural network can specifically be the size of the convolution kernel. Performing sparsity statistics on the image data can be to perform statistics on the valid data in the image data. The sparsity statistics result can include the positions and the number of valid data.

[0049] For example, the image data can be divided into multiple image blocks. For each image block, according to the size of the convolution kernel of the neural network, multiple data to be operated on are determined, and an OR operation is performed on each data to be operated on to determine whether the data to be operated on is valid data, so that the positions and the number of valid data in each image block can be statistically obtained. It can be understood that each image block corresponds to a sparsity statistics result.

[0050] In operation S230, based on the sparsity statistics result, valid data is determined from the image data.

[0051] The sparsity statistics result can include the positions and the number of valid data. Specifically, the image data can be divided into multiple image blocks, and each image block corresponds to a sparsity statistics result.

[0052] Determining valid data from the image data can be performed for each image block. Based on the number of valid data in the sparsity statistics result, it is determined whether there is valid data in the image block. If the number of valid data is zero, it indicates that there is no valid data in the image block. In this case, subsequent processing of this image block is no longer performed, and it continues to determine whether the number of valid data in the next image block is zero. If the number of valid data is not zero, the valid data can be determined based on the positions of the valid data.

[0053] In operation S240, the valid data is input into the spiking neural network for calculation to obtain multiple output results.

[0054] For example, the valid data can be a matrix in binary form extracted from the image data according to the sparsity statistics result. Inputting this matrix into the trained spiking neural network for calculation can obtain multiple output results.

[0055] For example, the trained spiking neural network can include a first spiking neural network and a second spiking neural network. The valid data can be input into the first spiking neural network. The first spiking neural network can include multiple spiking neural networks with different parameters. In this way, multiple different data can be obtained. Inputting the multiple different data into the second spiking neural network can obtain multiple output results. Similarly, the second spiking neural network can also include multiple spiking neural networks.

[0056] In operation S250, the multiple output results are fused to obtain a fused result.

[0057] For example, the multiple output results can be multiple binary data. The way to fuse the multiple output results can be to add these multiple binary data to obtain the fused result. Taking the computing task of recognizing numbers in an image as an example, after obtaining the fused result, the result can also be input into a fully connected layer to obtain the recognition result.

[0058] According to another embodiment of the present disclosure, the sparsity statistics result includes the positions and the number of valid data. Based on the convolution size of the trained spiking neural network, sparsity statistics are performed on the image data, and the obtained sparsity statistics result includes: dividing the image data into multiple image blocks based on the convolution size of the trained spiking neural network, extracting multiple data to be operated on from each image block; performing an OR operation on the multiple data to be operated on to obtain multiple OR operation results; in the case where the OR operation result is not zero, determining the data to be operated on as valid data; based on the multiple OR operation results, determining the positions and the number of valid data.

[0059] For example, according to the size of the convolution kernel of the spiking neural network, the image data can be divided into multiple image blocks. Taking a convolution kernel of 3×3 and the image data as an n*n binary matrix as an example, the image data can be divided into multiple 3*n image blocks. Extracting multiple data to be operated on from each image block can be, for each 3*n image block, extracting multiple 3×3 data to be operated on, and all the data to be operated on can be stored in the register array i1.

[0060] Performing an OR operation on each data to be operated on can be performing an OR operation on the 9 binary numbers in each 3×3 data to be operated on. If all these 9 binary numbers are zero, the result of the OR operation is 0, indicating that this data to be operated on is invalid data and does not need to participate in the operation. If these 9 binary numbers are not all zero, the result of the OR operation is 1. Storing all the operation results in the register array i2. In this way, the data stored in the register array i2 can represent the positions of the valid data.

[0061] According to the binary numbers with the operation result of 1 in the register array i2, the number of valid data in the image block can be obtained, and the number of valid data can be stored in the register array i3.

[0062] According to another embodiment of the present disclosure, based on the convolution size of the trained spiking neural network, dividing the image data into multiple image blocks, and extracting multiple data to be operated on from each image block includes: adding a zero element at the start and end of each row of data in the image data to obtain a first matrix; adding a zero element at the start and end of each column of data in the first matrix to obtain a second matrix; based on the convolution size of the spiking neural network, dividing the second matrix into multiple image blocks, and extracting multiple data to be operated on from each image block.

[0063] Taking the image data as an N*N matrix as an example, the image data can be represented as X1, X2, …, X N , where X1 represents the data of the first row in the image data, X2 represents the data of the second row in the image data, and X N represents the data of the Nth row in the image data. Padding the image data, specifically adding a zero element at the start and end of each row in X1, X2, …, X N to obtain a first matrix, and then adding a zero element at the start and end of each column in the first matrix to obtain a second matrix, that is, adding all-zero rows X0 and X N+1 .

[0064] Based on the convolution size of the spiking neural network, dividing the second matrix into multiple image blocks can refer to dividing the second matrix into multiple image blocks based on the size of the convolution kernel of the spiking neural network. For example, if the size of the convolution kernel is S*S, then S rows of elements are extracted sequentially each time, and these S rows of elements are an image block. Extracting multiple data to be operated on from each image block can refer to, based on the size of the convolution kernel being S*S, extracting multiple matrices of size S*S from each S rows of elements as the data to be operated on.

[0065] It can be understood that padding zero elements to the image data can maintain the dimension of the data and facilitate subsequent data processing.

[0066] According to another embodiment of the present disclosure, determining valid data from the image data based on the sparsity statistical result includes: for each image block, when the number of valid data is not zero, determining the valid data according to the positions of the valid data.

[0067] For example, the number of valid convolution kernels of each image block can be taken out from the register array i3 in sequence. If the result is 0, the processing of this image block is skipped. Otherwise, the OR operation result Q of each image block is taken out from the register array i2, and the operation Q&(-Q) is performed to obtain the operation result W. Here, -Q represents the complement of Q. According to the operation result W, the positions where valid data is needed are gradually extracted from the right side of W. The valid data is determined according to the positions of the valid data.

[0068] It can be understood that according to the embodiments of the present disclosure, whether there is valid data in an image block can be judged by the number of valid data in the image block. If not, the next image block can be directly processed. In this way, the calculation brought by redundant data can be reduced, the energy efficiency ratio and throughput rate of the calculation process can be improved, and the calculation efficiency of the entire neural network can be improved.

[0069] According to another embodiment of the present disclosure, inputting the valid data into the spiking neural network for calculation to obtain multiple output results includes: storing the valid data in the first buffer; inputting the valid data in the first buffer into the spiking neural network for calculation to obtain the output result; when new valid data is received while processing the valid data in the first buffer, storing the new valid data in the second buffer; when the processing of the valid data in the first buffer is completed, inputting the valid data in the second buffer into the spiking neural network for calculation to obtain the output result.

[0070] For example, after determining the valid data, the valid data can be calculated through a pipelined ping-pong calculation mode. Specifically, the valid data is stored in the first buffer, and then the valid data in the first buffer is input into the spiking neural network for calculation. At the same time, new valid data can be received. If new valid data is received while processing the valid data in the first buffer, the new valid data can be stored in the second buffer. After the processing of the valid data in the first buffer is completed, the valid data in the second buffer can be input into the spiking neural network.

[0071] In addition, when the buffer is not full, the data in the buffer is not processed. After the buffer is full, the data in the buffer is input into the spiking neural network.

[0072] It can be understood that adopting this pipelined ping-pong calculation method can achieve efficient parallel convolution processing and optimize the data flow scheduling, enabling the present disclosure to achieve efficient calculation under low-power conditions.

[0073] According to another embodiment of the present disclosure, it further includes: when new valid data is received while processing the valid data in the second buffer, storing the new valid data in the first buffer; when the processing of the valid data in the second buffer is completed, inputting the valid data in the first buffer into the spiking neural network for calculation to obtain an output result.

[0074] For example, if new valid data is received during the processing of the valid data in the second buffer, the new valid data can be stored in the first buffer. After waiting for the processing of the valid data in the second buffer to be completed, the valid data in the first buffer is processed.

[0075] In addition, when the buffer is not full, the valid data in the buffer is not processed. After the buffer is full, the valid data in the buffer is input into the spiking neural network.

[0076] It can be understood that through the data sparsification processing and pipelined ping-pong calculation mode proposed by the present disclosure, efficient parallel calculation can be directly performed in the storage unit, significantly reducing the data transfer energy consumption caused by the separation of storage and calculation, improving the neural network calculation efficiency, and being applicable to application scenarios such as low-power neural network calculation, real-time intelligent perception, and embedded artificial intelligence inference.

[0077] According to another embodiment of the present disclosure, a spiking neural network includes a first spiking neural network and a second spiking neural network. Inputting valid data into the spiking neural network for calculation to obtain multiple output results includes: inputting the valid data into the first spiking neural network to obtain multiple data to be input. The first spiking neural network includes multiple different spiking neural networks; inputting the multiple data to be input into the second spiking neural network to obtain multiple output results.

[0078] For example, the valid data can be input into multiple spiking neural networks with different trained parameters, which can also be referred to as inputting the valid data into multiple different channels, where each channel involves different parameters, to obtain multiple data to be input. Then, inputting the multiple data to be input into the second spiking neural network to obtain multiple output results. The parameters of different channels can be the results of training the spiking neural network.

[0079] In addition, the method for extracting valid data from image data proposed in the present disclosure can also be used to extract valid data from multiple data to be input, and input the valid data in the data to be input into the second spiking neural network.

[0080] Taking the valid data as a 32×32 matrix and the first spiking neural network including three different spiking neural networks as an example, inputting the 32×32 matrix into 3 different spiking neural networks can obtain 3 16×16 matrices, which can also be referred to as 3 16×16 images. Then, inputting these 3 16×16 matrices into the second spiking neural network. Among them, the second spiking neural network can also include 3 spiking neural networks. For each spiking neural network, 3 16×16 matrices can be obtained.

[0081] Figure 3 Schematically shows a flowchart of an image processing method according to another embodiment of the present disclosure.

[0082] As Figure 3 shown, the image processing method of this embodiment includes operations S310 to S360

[0083] In operation S310, image data is acquired and classified row by row.

[0084] The image data can be environmental image data collected by an image acquisition device. Specifically, the image acquisition device can be a configurable parameter spiking image sensor, and the time, gain, and frame rate of image acquisition can be configured.

[0085] Classifying the image data row by row can refer to dividing the image data into X1, X2, …, X N , where X1 represents the data of the first row in the image data, X2 represents the data of the second row in the image data, X NRepresents the data of the Nth row in the image data.

[0086] In operation S320, zero elements are filled in the image data, and the image data is divided into multiple image blocks.

[0087] Filling zero elements in the image data specifically means adding a zero element to the start and end of each row of data in X1, X2, …, X N and adding all-zero rows X0 and X N+1 .

[0088] Dividing the image data into multiple image blocks can be done by dividing the image data obtained after filling zero elements according to the size of the convolutional kernel of the neural network. For example, if the size of the image data obtained after filling zero elements is (N + 2) * (N + 2) and the size of the convolutional kernel is 3×3, then the image data obtained after filling zero elements can be divided into multiple image blocks of size 3 * (N + 2). It can also be done by sequentially extracting 3 rows of elements in the order of X0, X1, X2, …, X N 、X N+1 , and each 3 rows of elements is an image block.

[0089] In operation S330, sparsity statistics are performed on each image block to obtain sparsity statistics results.

[0090] The sparsity statistics results can include the positions and numbers of valid data. For example, performing sparsity statistics on each image block can be done by determining multiple data to be operated on according to the size of the convolutional kernel of the neural network, that is, the input of the convolutional kernel. For example, if the size of the convolutional kernel is 3×3, each image block can be divided into multiple 3×3 data to be operated on, and then the binary numbers in each data to be operated on are ORed. If the OR operation result is 1, then the input is valid data, otherwise it is invalid data. In this way, the positions and numbers of valid data in each image block can be obtained.

[0091] In operation S340, valid data in the image data is determined based on the sparsity statistics results.

[0092] The sparsity statistics results can include the positions and numbers of valid data. Determining valid data in the image data based on the sparsity statistics results can be done by first judging whether the number of valid data in each image block is zero. If the number of valid data is not zero, then the valid data in the image block can be determined according to the positions of the valid data. If the number of valid data is zero, continue to process the next image block.

[0093] In operation S350, the valid data is input into a spiking neural network for calculation to obtain multiple output results.

[0094] The spiking neural network can be a trained spiking neural network with multiple channels, or it can be understood that each channel corresponds to a trained spiking neural network, and the parameters of the neural networks corresponding to each channel are different. For example, valid data can be input into multiple channels of the first spiking neural network to obtain multiple data to be input. It can be understood that the valid data received by each channel is the same, but due to the different parameters corresponding to each channel, the data to be input corresponding to each channel is different. Then, the multiple data to be input are input into the second spiking neural network, and multiple different output results can be obtained. It can be understood that the second spiking neural network can also have multiple different channels. Fusing the output results can obtain the fused result.

[0095] It can be understood that before inputting the multiple data to be input into the second spiking neural network, the method for extracting valid data from image data proposed in this disclosure can be used to extract valid data from the multiple data to be input, and the valid data in the data to be input is input into the second spiking neural network. In this way, the efficiency of the algorithm can be further improved.

[0096] In addition, inputting valid data into the spiking neural network for calculation and extracting valid data can be carried out simultaneously. This disclosure adopts a pipelined ping-pong operation mode. For example, the received valid data can be first stored in the first buffer, and then the valid data in the first buffer is input into the spiking neural network for operation. At the same time, the newly received valid data is stored in the second buffer. After the operation of the valid data in the first buffer is completed, if the second buffer is also filled with valid data, the valid data in the second buffer is operated on. If new valid data is received at this time, the new valid data is stored in the first buffer again, and this step is repeated until no new valid data is received.

[0097] In operation S360, multiple output results are fused to obtain the fused result.

[0098] Fusing the output results can mean adding multiple output results to obtain the fused result. After obtaining the fused result, the fused result can also be input into the fully connected layer to obtain the calculation result corresponding to the calculation task. For example, if the calculation task is to recognize the numbers in an image, then inputting the fused result into the fully connected layer can obtain the recognition result, that is, the numbers in the image.

[0099] Figure 4 Schematically shows a structural block diagram of a neural network model suitable for implementing an image processing method according to an embodiment of the present disclosure.

[0100] As Figure 4As shown, taking a spiking neural network including a first spiking neural network and a second spiking neural network, and the computing task being to identify the numbers in an image as an example. The valid data is sequentially input into the first spiking neural network and the second spiking neural network to obtain multiple output results, and then the multiple output results are input into a fully connected layer to obtain the final recognition result.

[0101] Figure 5 Schematically shows a structural block diagram of a neural network model suitable for implementing an image processing method according to another embodiment of the present disclosure.

[0102] As Figure 5 shown, taking the first spiking neural network including three different channels (i.e., including three different spiking neural networks), and the valid data being a 32×32 image as an example, the valid data is input into the three different channels, and each channel corresponds to a convolutional kernel with different parameters. Then, the output results of these three channels are respectively input into the corresponding pooling layer to obtain the data to be input for each channel, specifically a 16×16 matrix, which can also be called a 16×16 image.

[0103] It can be understood that the valid data input for each channel is the same, but due to different parameters such as the weights of the neural networks involved in each channel, the output results of these three channels are different, and the specific parameters depend on the results of training the neural network.

[0104] Figure 6 Schematically shows a structural block diagram of a neural network model suitable for implementing an image processing method according to another embodiment of the present disclosure.

[0105] As Figure 6 shown, taking the second spiking neural network including three channels as an example, the output of the first spiking neural network is input into the three channels of the second spiking neural network in a graphical manner. For each channel, three 16×16 matrices can be obtained. In the fusion and addition module, for each channel, the three 16×16 matrices are added and input into the corresponding pooling layer, and finally three 8×8 images are obtained. These three 8×8 images can be used as the input for the fully connected layer.

[0106] In addition, before inputting the output result of the first spiking neural network into the second spiking neural network, the output result of the first spiking neural network can be statistically analyzed for sparsity again to obtain a sparsity statistical result, and based on the sparsity statistical result, valid data is extracted, and this valid data is used as the input for the second spiking neural network.

[0107] It can be understood that for the three channels in the second pulse neural network, there are three convolutional kernels with the same parameters in each channel, corresponding to the three different inputs received by each channel (i.e., the output of the first pulse neural network). However, the convolutional kernels of different channels are different. Therefore, the outputs of the three channels of the second pulse neural network are also different.

[0108] Figure 7 Schematically shows a structural block diagram of a neural network model suitable for implementing an image processing method according to another embodiment of the present disclosure.

[0109] As Figure 7 shown, taking the full connection input as 3 images of 8×8 and the calculation task as recognizing the numbers in the images as an example. Inputting these 3 images of 8×8 into the full connection layer can obtain the recognition result, and the specific recognition result is the numbers in the images.

[0110] Figure 8 Schematically shows a structural block diagram of an image processing system according to an embodiment of the present disclosure.

[0111] As Figure 8 shown, the image processing system 800 may be an SNN image intelligent processing system based on a sense-compute integrated architecture. The image processing system 800 includes a pulsed image sensor 810, a data storage module 820, a data control module 830, and an SNN acceleration module 840. Among them, the SNN acceleration module 840 may specifically include a plurality of single-channel acceleration modules 841, a memory-compute integrated computing array 842, and a multi-channel fusion module 843.

[0112] The pulsed image sensor 810 is connected to the data control module 830, the data storage module 820 is connected to the data control module 830, and the SNN acceleration module 840 is connected to the data control module 830.

[0113] The pulsed image sensor 810 is used to collect environmental images in real time. For example, the pulsed image sensor 810 adopts a high-speed pulsed image sensor with configurable parameters. The exposure time, gain, and frame rate of the image output are configured through the data control module 830. The pulsed image sensor 810 performs continuous imaging in a global exposure manner. After imaging is completed, the serial image data is output pixel by pixel through a serial interface and sent to the data control module 830.

[0114] The data storage module 820 is used to store image data and intermediate calculation results to support the calculation requirements of the SNN acceleration module. For example, the data storage module 820 can adopt a hierarchical structure, which is divided into multiple groups of storage units. Each group of storage units corresponds to a group of calculation units, enabling parallel access to data and improving the throughput rate between data storage and calculation. For example, the data storage module 820 can be divided into two groups. Specifically, the fourth-generation double data rate synchronous dynamic random access memory (DDR4) can be used to optimize data caching and storage efficiency. The first group of the data storage module can be responsible for caching the original images output by the pulsed image sensor 810. The real-time images captured by the current pulsed image sensor can be obtained through the first group of the data storage module at any time. The second group of the data storage module is responsible for storing the original image data to be processed. After the image processing is completed, the processed results after calculation can be obtained through the second group of the data storage module

[0115] The data control module 830 is used to coordinate image acquisition, data storage, and SNN calculation tasks to ensure the computational efficiency of the entire system. Specifically, it can be used to manage the pulsed image sensor 810, the data storage module 820, and the data stream, and can configure the exposure time, gain, and frame rate of the pulsed image sensor 810 to acquire image data

[0116] For example, the data control module 830 can send calculation tasks to the SNN acceleration module 840 and read the input data required for calculation from the data storage module 820. The data control module 830 can also send a data feedback instruction to the data storage module 820 to ensure the correct storage of the calculation results. If the calculation results need further processing, the data control module 830 can transmit the calculation results to other calculation nodes or external systems

[0117] The SNN acceleration module 840 is used to execute the calculations of the spiking neural network. Specifically, it can be a SNN acceleration calculation module based on the memory-computation integrated structure. The SNN acceleration module adopts a structure combining multiple single-channel acceleration modules 841, a memory-computation integrated calculation array 842, and a multi-channel fusion module 843. The SNN acceleration module 840 can reduce redundant calculations through the optimized data scheduling mechanism proposed in this disclosure

[0118] The single-channel acceleration module 841 is used to execute single-channel SNN calculations. In the single-channel acceleration module 841, the sparsity of data can be utilized to accelerate the SNN for a single channel. Specifically, the valid data can be extracted through the method proposed in this disclosure in the single-channel acceleration module 841, and the valid data can be input into the memory-computation integrated array

[0119] The in-memory computing array 842 is used to perform efficient parallel computing.

[0120] The multi-channel fusion module 843 is used to fuse the SNN calculation results of multiple channels. The multi-channel fusion module 843 is connected to multiple single-channel acceleration modules 841, used to fuse the calculation results of each channel, and input the fused results into the fully connected layer to obtain the final result, that is, the result corresponding to the calculation task. This result can be stored in the data storage module 820 and transmitted to the external interface through the data control module 830 for application.

[0121] According to the image processing method and image processing system proposed in the present disclosure, after the pulsed image sensor 810 acquires an image, it sends a data write request to the data control module 830, and the data control module 830 writes the image data into the data storage module 820; at the same time, the data control module 830 can send a data read request to the SNN acceleration module 840 according to the calculation requirements, and the SNN acceleration module 840 retrieves the image data from the data storage module 820 and performs calculations based on the pulsed neural network. Specifically, the single-channel acceleration module 841 extracts the valid data by classifying the image data row by row, filling zero elements, dividing into multiple image blocks, and sparsity statistics, and inputs the valid data into the in-memory computing array 842 for parallel operation, specifically using the pipelined ping-pong calculation method proposed in this application. After the calculation is completed, the SNN acceleration module 840 transmits the processing result to the data control module 830 or stores it in the data storage module 820 to achieve low-power and real-time intelligent image processing based on the sensor-in-memory integrated architecture.

[0122] In addition, the calculation result can be transmitted to the host computer, cloud server or display terminal through the data control module 830 for further analysis or storage. If the image processing system 800 needs to perform loop calculations, the data storage module 820 can use the result as input for subsequent calculation tasks. If the calculation result involves classification, detection or recognition tasks, the final decision result can be sent to the control system for response.

[0123] The entire system can be implemented using a programmable logic device without an operating system, and can complete pulsed image acquisition, preprocessing, SNN calculation and result output at the edge.

[0124] It can be understood that, at the hardware level, the present disclosure adopts a computing-in-memory integrated computing architecture, combined with an SNN computing optimization strategy, which can improve the energy efficiency ratio of neural network computing. Specifically, the SNN image intelligent processing system based on the computing-in-memory integrated architecture proposed by the present disclosure integrates the pulsed image sensor 810, the data storage module 820, the data control module 830, and the SNN acceleration module 840 into the same system. By adopting the computing-in-memory integrated computing architecture, data can be directly calculated in the storage unit, reducing the energy consumption caused by data transmission, improving the computing efficiency, and being applicable to low-power real-time computing scenarios. The SNN acceleration module 840 of the present disclosure extracts valid data through the single-channel acceleration module 841 and inputs it into the computing-in-memory computing array 842 for calculation, significantly reducing the redundant calculation caused by invalid data. The computing-in-memory computing array 842 can implement efficient parallel convolution operations on valid data. Through the multi-channel fusion module 843, the SNN calculation results of different channels can be fused to meet the multi-channel parallel inference requirements. Through technical means such as data flow scheduling optimization and SNN sparse calculation, the energy efficiency ratio and throughput rate of the computing process are improved, ensuring that the present system is applicable to real-time intelligent perception tasks.

[0125] Figure 9 Schematically shows a structural block diagram of an image processing system according to another embodiment of the present disclosure.

[0126] As Figure 9 shown, the data control module 930 includes a microcontroller 931, a data bus 932, an acceleration module control component 933, a data storage control component 934, and a pulsed image sensor control component 935. Among them, the acceleration module control component 933 is connected to the SNN acceleration module 940, the data storage control component 934 is connected to the data storage module 920, and the pulsed image sensor control component 935 is connected to the image sensor 910.

[0127] The microcontroller 931 is sequentially connected to the data bus 932, the pulsed image sensor control component 935, the data storage control component 934, and the acceleration module control component 933, and is used to manage system control signals. For example, the microcontroller 931 can control the topology of the SNN neural network of the SNN acceleration module 940 through the acceleration module control component 933, and can also control the data writing and reading of the data storage module 920 through the data storage control component 934, and can also configure the exposure time, gain, and frame rate of the pulsed image sensor 910 to collect images through the pulsed image sensor control component 935.

[0128] The data bus 932 is used to transfer data between modules. Specifically, the data bus 932 is sequentially connected to the data storage control component 934, the microcontroller 931, the SNN acceleration module 940, and the data memory 920, and is used to control the data transfer of the entire system.

[0129] The acceleration module control component 933 is sequentially connected to the microcontroller 931 and the SNN acceleration module 940, and is used for the control of the SNN acceleration module 940.

[0130] The data storage control component 934 is sequentially connected to the microcontroller 931, the data bus 932, and the data memory module 920, and is specifically used to manage the data reading and writing of the data storage module 920.

[0131] The pulse image sensor control component 935 is used to control the parameter configuration of the pulse image sensor 910. The pulse image sensor 910 can write data to the data bus 932 through the pulse image sensor control component 935.

[0132] Figure 10 The structural block diagram of an image processing system according to another embodiment of the present disclosure is schematically shown.

[0133] As Figure 10 shown, the in-memory computing array 1042 includes a ping-pong buffer controller 10421, a first buffer 10422, a second buffer 10423, and an in-memory computing array core 10424.

[0134] The in-memory computing array 1042 receives the valid data from the single-channel acceleration module, stores all the received valid data into the ping-pong buffer 10421, and then stores it into the first buffer 10422 until the start operation flag signal is received. The start operation flag signal can be sent by the single-channel acceleration module after fetching all the valid data.

[0135] The in-memory computing array core 10424 performs operations on the valid data. Specifically, the valid data can be input into a pulsed neural network for operations. While performing operations, it can receive the valid data from the single-channel acceleration module to achieve pipelining control. At this time, the valid data from the single-channel acceleration module is stored into the ping-pong buffer controller 10421, and then stored into the second buffer 10423 to achieve ping-pong buffering operation. After the in-memory computing array core 10424 completes the operation, the result can be returned to the corresponding single-channel acceleration module.

[0136] After the data operation in the first buffer 10422 is completed, if the second buffer 10423 is not yet full of valid data to be calculated, the in-memory computing array core 10424 waits for the second buffer 10423 to be full, and repeats the above steps after the second buffer 10423 is full. It can be understood that this method can achieve efficient parallel convolution processing.

[0137] Figure 11 The structural block diagram of the image processing device according to an embodiment of the present disclosure is schematically shown.

[0138] As Figure 11 shown, the image processing device 1100 of this embodiment includes an acquisition module 1110, a statistics module 1120, a determination module 1130, a calculation module 1140, and a fusion module 1150.

[0139] The acquisition module 1110 is used to acquire image data. In one embodiment, the acquisition module 1110 can be used to perform the operation S210 described above, which will not be elaborated here.

[0140] The statistics module 1120 is used to perform sparsity statistics on the image data based on the convolution size of the trained spiking neural network to obtain a sparsity statistics result. In one embodiment, the statistics module 1120 can be used to perform the operation S220 described above, which will not be elaborated here.

[0141] The determination module 1130 is used to determine valid data from the image data based on the sparsity statistics result. In one embodiment, the determination module 1130 can be used to perform the operation S230 described above, which will not be elaborated here.

[0142] The calculation module 1140 is used to input the valid data into the spiking neural network for calculation to obtain multiple output results. In one embodiment, the calculation module 1140 can be used to perform the operation S240 described above, which will not be elaborated here.

[0143] The fusion module 1150 is used to fuse multiple output results to obtain a fused result. In one embodiment, the fusion module 1150 can be used to perform the operation S250 described above, which will not be elaborated here.

[0144] Figure 12 The schematic block diagram of the electronic device that can be used to implement the image processing method according to an embodiment of the present disclosure is schematically shown.

[0145] As Figure 12As shown, the electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage section 1208 into a random access memory (RAM) 1203. The processor 1201 can include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application-specific integrated circuit (ASIC)), etc. The processor 1201 can also include on-board memory for caching purposes. The processor 1201 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0146] In the RAM 1203, various programs and data required for the operation of the electronic device 1200 are stored. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. The processor 1201 performs various operations of the method flow according to an embodiment of the present disclosure by executing the programs in the ROM 1202 and / or the RAM 1203. It should be noted that the program can also be stored in one or more memories other than the ROM 1202 and the RAM 1203. The processor 1201 can also implement the image processing method provided by the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0147] According to an embodiment of the present disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, and the input / output (I / O) interface 1205 is also connected to the bus 1204. The electronic device 1200 may further include one or more of the following components connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read from it can be installed into the storage section 1208 as needed.

[0148] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist alone without being assembled into the device / apparatus / system. The above computer-readable storage medium stores one or more programs, and when the one or more programs are executed, the methods according to the embodiments of the present disclosure are implemented.

[0149] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the above-described ROM 1202 and / or RAM 1203 and / or one or more memories other than ROM 1202 and RAM 1203.

[0150] An embodiment of the present disclosure also includes a computer program product, which includes a computer program that contains program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to cause the computer system to implement the method provided by the embodiments of the present disclosure.

[0151] When the computer program is executed by the processor 1201, the above functions defined in the system / apparatus of the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the above-described systems, apparatuses, modules, units, etc. may be implemented by computer program modules.

[0152] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and is downloaded and installed through the communication part 1209, and / or installed from the removable medium 1211. The program code included in the computer program may be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0153] In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above functions defined in the system of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the above-described systems, devices, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0154] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application, etc. of the user's personal information involved all comply with the provisions of relevant laws and regulations, adopt necessary confidentiality measures, and do not violate public order and good customs. In the technical solution of the present disclosure, the authorization or consent of the user is obtained before obtaining or collecting the user's personal information.

[0155] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedures and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include but are not limited to, for example, Java, C++, Python, the "C" language, or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by connecting through the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0157] Those skilled in the art will appreciate that the features recited in the various embodiments and / or claims of the present disclosure may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly recited in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features recited in the various embodiments and / or claims of the present disclosure may be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present disclosure.

[0158] The embodiments of the present disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments have been described separately above, this does not mean that the measures in the respective embodiments cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present disclosure.

Claims

1. An image processing method, characterized in that, The method includes: Obtaining image data; Based on the convolution size of the trained spiking neural network, performing sparsity statistics on the image data to obtain a sparsity statistics result; Based on the sparsity statistics result, determining valid data from the image data; Inputting the valid data into the spiking neural network for calculation to obtain multiple output results; Fusing the multiple output results to obtain a fused result.

2. The method according to claim 1, characterized in that, The sparsity statistics result includes the positions and the number of valid data. The performing sparsity statistics on the image data based on the convolution size of the trained spiking neural network to obtain a sparsity statistics result includes: Based on the convolution size of the trained spiking neural network, dividing the image data into multiple image blocks, and extracting multiple data to be operated on from each image block; Performing an OR operation on the multiple data to be operated on to obtain multiple OR operation results; When the OR operation result is not zero, determining the data to be operated on as valid data; Based on the multiple OR operation results, determining the positions and the number of valid data.

3. The method according to claim 2, wherein The dividing the image data into multiple image blocks based on the convolution size of the trained spiking neural network and extracting multiple data to be operated on from each image block includes: Adding a zero element at each of the start and end of each row of data in the image data to obtain a first matrix; Adding a zero element at each of the start and end of each column of data in the first matrix to obtain a second matrix; Based on the convolution size of the spiking neural network, dividing the second matrix into multiple image blocks, and extracting multiple data to be operated on from each image block.

4. The method according to claim 3, wherein The determining valid data from the image data based on the sparsity statistics result includes: For each image block, when the number of valid data is not zero, determining the valid data according to the positions of the valid data.

5. The method according to claim 1, wherein The inputting the valid data into the spiking neural network for calculation to obtain multiple output results includes: Storing the valid data into a first buffer; Inputting the valid data in the first buffer into the spiking neural network for calculation to obtain an output result; When new valid data is received while processing the valid data in the first buffer, storing the new valid data into a second buffer; When the processing of the valid data in the first buffer is completed, inputting the valid data in the second buffer into the spiking neural network for calculation to obtain an output result.

6. The method according to claim 5, wherein The method further includes: When new valid data is received while processing the valid data in the second buffer, storing the new valid data into the first buffer; When the processing of the valid data in the second buffer is completed, inputting the valid data in the first buffer into the spiking neural network for calculation to obtain an output result.

7. The method according to claim 1, characterized in that The spiking neural network includes a first spiking neural network and a second spiking neural network. The inputting the valid data into the spiking neural network for calculation to obtain multiple output results includes: Inputting the valid data into the first spiking neural network to obtain multiple data to be input, and the first spiking neural network includes multiple different spiking neural networks; Input multiple data to be input into the second spiking neural network to obtain multiple output results.

8. An image processing apparatus, characterized in that, The device includes: An acquisition module, configured to acquire image data; A statistics module, configured to perform sparsity statistics on the image data based on the convolution size of the trained spiking neural network to obtain a sparsity statistics result; A determination module, configured to determine valid data from the image data based on the sparsity statistics result; A calculation module, configured to input the valid data into the spiking neural network for calculation to obtain multiple output results; A fusion module, configured to fuse multiple output results to obtain a fused result.

9. An electronic device, comprising: One or more processors; A storage device, configured to store one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, having executable instructions stored thereon, which when executed by a processor cause the processor to execute the method according to any one of claims 1 to 7.

Images