Visual network processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a visual network processing method and device, electronic equipment and a storage medium. The method comprises the steps of determining a task type of a to-be-processed task of a visual network; if the task type is a visual task type, the visual network is split to obtain a lower sampling subnet, a feature space conversion subnet and an upper sampling subnet, and the lower sampling subnet is used for performing down-sampling processing, blocking processing and block fusion processing on the input image; the feature space conversion subnet is used for performing feature extraction processing and feature dimension conversion processing on the output image of the lower sampling subnet; and the upper sampling subnet is used for carrying out upper sampling reduction processing on the output image of the feature space conversion subnet and executing a to-be-processed task on the reduced image. According to the invention, multiple memory exchange operations caused by insufficient memory during bulk data carrying can be reduced, and the processing efficiency of a visual network for executing a visual task is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a method, device, electronic device and storage medium for processing a visual network. Background Art

[0002] At present, artificial intelligence technology has shown a significant trend towards general models. For example, it has performed well in the fields of natural language processing and large language models. Similar research has also gradually appeared in the field of computer vision. However, artificial intelligence visual networks are inefficient when performing visual tasks. Therefore, how to improve the efficiency of visual networks in performing visual tasks has become an urgent problem to be solved. Summary of the invention

[0003] In view of the above problems, the present application proposes a visual network processing method, device, electronic device and storage medium to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a method for processing a visual network, the method comprising: determining a task type of a task to be processed of the visual network; if the task type is a preset visual task type, splitting the visual network to obtain a downsampling subnet, a feature space conversion subnet and an upsampling subnet, wherein: the downsampling subnet is used to perform downsampling, block processing and block fusion processing on the input image; the feature space conversion subnet is used to perform feature extraction processing and feature dimension conversion processing on the output image of the downsampling subnet; the upsampling subnet is used to perform upsampling and restoration processing on the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

[0005] In a second aspect, an embodiment of the present application provides a processing device for a visual network, the device comprising: a task type determination module, used to determine the task type of a task to be processed of the visual network; a splitting module, used to split the visual network if the task type is a preset visual task type, to obtain a downsampling subnet, a feature space conversion subnet and an upsampling subnet, wherein: the downsampling subnet is used to downsample, block and block fusion the input image; the feature space conversion subnet is used to perform feature extraction and feature dimension conversion on the output image of the downsampling subnet; the upsampling subnet is used to upsample and restore the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0008] The processing method, device, electronic device and storage medium of the visual network provided in the embodiments of the present application determine whether to split the visual network according to the task type of the to-be-processed task of the determined visual network, and when the task type is a preset visual task type, split the visual network into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, so that the input image is downsampled, blocked and fused by the downsampling subnet, and then the feature space conversion subnet performs feature extraction and feature dimension conversion on the output image of the downsampling subnet, and finally the upsampling subnet performs upsampling and restoration on the output image of the feature space conversion subnet, and executes the to-be-processed task on the restored image, thereby reducing multiple memory exchange operations caused by insufficient memory when moving large blocks of data, and improving the processing efficiency of the visual network in executing preset visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of a process flow of a visual network processing method provided by an embodiment of the present application is shown;

[0011] Figure 2 A schematic diagram of a process flow of a visual network processing method provided by an embodiment of the present application is shown;

[0012] Figure 3 A schematic diagram showing a downsampling subnet according to an embodiment of the present application performing block processing and block fusion processing on an input image;

[0013] Figure 4 A schematic diagram showing a feature space conversion subnet provided in an embodiment of the present application performing feature dimension conversion processing on an output image of a downsampling subnet;

[0014] Figure 5 A schematic diagram showing a process flow of a visual network processing method provided in an embodiment of the present application is shown;

[0015] Figure 6 A schematic diagram of a process flow of a visual network processing method provided by an embodiment of the present application is shown;

[0016] Figure 7 A schematic diagram of a target subgraph provided in an embodiment of the present application is shown;

[0017] Figure 8 Performance diagram of HMX for reasoning based on a visual network of a target subgraph provided in an embodiment of the present application;

[0018] Fig. 9 Performance graph of HMX for reasoning with a visual network that deletes a target subgraph provided in an embodiment of the present application;

[0019] Fig.10 A schematic diagram of a process flow of a visual network processing method provided by an embodiment of the present application is shown;

[0020] Fig.11 A schematic diagram of a target operator provided in an embodiment of the present application;

[0021] Fig.12 A schematic diagram of a process flow of a visual network processing method provided by an embodiment of the present application is shown;

[0022] Fig.13 A schematic diagram of a combinatorial operator provided in an embodiment of the present application;

[0023] Fig.14 Performance diagram of HMX based on zero-filling convolution processing provided in the embodiment of the present application;

[0024] Fig.15 Performance diagram of HMX based on convolution processing using a reflection filling method provided in an embodiment of the present application;

[0025] Fig.16 A module block diagram of a visual network processing device provided by an embodiment of the present application is shown;

[0026] Fig.17 A block diagram of an electronic device for executing a startup abnormality processing method according to an embodiment of the present application is shown;

[0027] Fig.18 A storage unit for storing or carrying program codes for implementing a method for handling power-on exceptions according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0029] In response to the above problems, the inventors have discovered after long-term research and proposed the processing method, device, electronic device and storage medium of the visual network provided in the embodiments of the present application. By determining whether to split the visual network according to the task type of the task to be processed of the determined visual network, and when the task type is a visual task type, the visual network is split into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, so that the input image is downsampled, block-processed and block-fused by the downsampling subnet, and then the feature space conversion subnet performs feature extraction and feature dimension conversion on the output image of the downsampling subnet, and finally the upsampling subnet performs upsampling and restoration on the output image of the feature space conversion subnet, and performs the task to be processed on the restored image, thereby reducing the multiple memory exchange operations caused by insufficient memory when moving large blocks of data, and improving the processing efficiency of the visual network in performing visual tasks. Among them, the specific processing method of the visual network is described in detail in the subsequent embodiments.

[0030] See also Figure 1 , Figure 1 A flow chart of a method for processing a visual network provided by an embodiment of the present application is shown. The method is used to determine whether to split the visual network according to the task type of the task to be processed of the determined visual network, and when the task type is a visual task type, split the visual network into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, so that the input image is downsampled, blocked and fused by the downsampling subnet, and then the feature space conversion subnet performs feature extraction and feature dimension conversion on the output image of the downsampling subnet, and finally the upsampling subnet performs upsampling and restoration on the output image of the feature space conversion subnet, and executes the task to be processed on the restored image, thereby reducing multiple memory exchange operations caused by insufficient memory when moving large blocks of data, and improving the processing efficiency of the visual network in performing visual tasks. In a specific embodiment, the method for processing a visual network is applied to Fig.17 The visual network processing device 200 and the electronic device 100 equipped with the visual network processing device 200 are shown. Fig.18 ), the processing method of the visual network can be applied to multiple scenarios, for example, it can be applied to the visual processing task of group photos, and also to the application of super-resolution network, image restoration tasks, etc., which will not be elaborated here. The following will take electronic devices as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic devices used in this embodiment may include smart phones, tablet computers, wearable electronic devices, etc., which are not limited here. Figure 1 The process shown is described in detail, and the processing method of the visual network may specifically include the following steps:

[0031] Step S110: Determine the task type of the task to be processed in the visual network.

[0032] In some embodiments, the visual network may be a neural network used to perform visual tasks, and the visual network may include a convolutional neural network and / or a fully connected neural network. The visual network may be a network deployed locally in the electronic device, and the user may perform different visual tasks through the visual network deployed locally in the electronic device. Of course, the visual network may also be deployed in a server that communicates with the electronic device, which is not limited here.

[0033] Optionally, the task type of the task to be processed by the visual network can be determined by obtaining an input image of an electronic device deployed with the visual network and determining a corresponding task identifier in the input image, wherein the visual network can perform a corresponding visual task according to the corresponding task identifier in the input image.

[0034] Optionally, the task types of the tasks to be processed may include visual tasks, image classification tasks, image localization tasks, image detection tasks, and image segmentation tasks, etc. Visual tasks of different task types may perform different calculations in the visual network.

[0035] Step S120: If the task type is a preset visual task type, the visual network is split to obtain a downsampling subnet, a feature space conversion subnet and an upsampling subnet. The downsampling subnet is used to perform downsampling, block processing and block fusion processing on the input image, the feature space conversion subnet is used to perform feature extraction and feature dimension conversion processing on the output image of the downsampling subnet, and the upsampling subnet is used to perform upsampling and restoration processing on the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

[0036] In some embodiments, the preset visual task type may be a visual task having at least one characteristic of a deep number of network layers for processing, a large receptive field, and the ability to globally calculate statistical information. For example, the preset visual processing task type may be a bottom-level visual processing task type. The processing performed by the visual network when executing a visual task of the preset visual task type is different from that performed when executing other visual tasks of non-preset visual task types. The tasks of the preset visual task type require the calculation of global information. Therefore, the features extracted by the deep convolution of the visual network are obtained after multiple convolution operations, and the extracted features are closer to the output layer of the visual network, adding more coarse granularity and more abstract semantic information. In order to obtain information about the integrity of the image, the receptive field of the image is increased, and the image information is increased, so that the visual network maintains the characteristics of a deep number of network layers, a large receptive field, and the ability to globally calculate statistical information. However, these characteristics will increase the memory usage of the electronic device, and then lead to multiple memory exchange operations, which will lead to the inefficiency of the visual network when performing tasks of the visual task type and the problem of insufficient memory.

[0037] Therefore, this embodiment proposes to split the visual network based on the different natures of the steps processed by different subnets in the visual network to obtain a downsampling subnet, a feature space conversion subnet and an upsampling subnet, so as to perform different operations through the downsampling subnet, the feature space conversion subnet and the upsampling subnet, thereby improving the efficiency of the visual network in performing tasks of preset visual task types.

[0038] Optionally, the first few convolutional layers in the visual network usually perform downsampling operations with a convolution kernel moving step size greater than 1 to obtain a smaller feature map size, and then perform multiple feature dimension conversions on the smaller size. Finally, upsampling is performed through deconvolution or interpolation to restore the original resolution to perform the corresponding visual task. Furthermore, the visual network can be split based on the operational characteristics of the visual network when performing visual tasks, and the network used to perform feature dimension conversion and feature extraction is used as a feature space conversion subnet, and the network used to perform downsampling operations is used as a downsampling subnet, and the network used to perform upsampling operations is used as an upsampling subnet, so as to split the visual network.

[0039] Optionally, since visual tasks usually cannot damage the image quality, in order to ensure that the preset visual tasks can be performed, the input size should be kept large and cannot be displayed as decreased during the process of the visual network executing the preset visual tasks to process the input image. However, a larger input size will take up more memory, and the memory and computing power of the device deployed with the visual network are limited. The parameters and data of the visual network are difficult to store in the memory at the same time, which will inevitably lead to additional memory exchange and loading operations, thereby increasing the time and power consumption of reasoning. Therefore, this embodiment divides the input image into blocks through the downsampling subnet after the visual network is split, so that the image information corresponding to each block after the block can be perceived. Since the size of the image block after the block is small, memory exchange can be avoided, and the computing power resources of the electronic device can be fully utilized to improve the efficiency of the visual network in performing the preset visual tasks.

[0040] Optionally, since the image information perceived by the downsampling subnet is the image information corresponding to each image block after fusion processing, the dimension of the corresponding information is relatively low. Therefore, the feature space conversion subnet after the visual network split can be used to perform feature dimension conversion and feature extraction on the image output by the downsampling subnet.

[0041] Optionally, since the initial feature map is obtained by processing the down-sampled image through the feature space conversion subnet, and the particularity of the preset visual task usually requires global information, in order to restore the size corresponding to the input image to obtain the global information, the upsampling subnet after splitting the visual network can be used for restoration processing, and then the corresponding task to be processed can be executed based on the restored feature map, that is, the preset visual task can be executed.

[0042] In an embodiment of the present application, whether to split the visual network is determined according to the task type of the pending task of the determined visual network, and when the task type is a preset visual task type, the visual network is split into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, so that the input image is downsampled, blocked and fused by the downsampling subnet, and then the output image of the downsampling subnet is subjected to feature extraction and feature dimension conversion by the feature space conversion subnet, and finally the output image of the feature space conversion subnet is upsampled and restored by the upsampling subnet, and the pending task is executed on the restored image, thereby reducing multiple memory exchange operations caused by insufficient memory when moving large blocks of data, and improving the processing efficiency of the visual network in executing preset visual tasks.

[0043] See also Figure 2 , Figure 2 The flowchart of the processing method of the visual network provided by an embodiment of the present application is shown in FIG. Figure 2The process shown is described in detail, and the processing method of the visual network may specifically include the following steps:

[0044] Step S210: Determine the task type of the task to be processed in the visual network.

[0045] Step S220: If the task type is a preset visual task type, the visual network is split into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, wherein: the downsampling subnet is used to downsample, block and fusion the input image; the feature space conversion subnet is used to perform feature extraction and feature dimension conversion on the output image of the downsampling subnet; the upsampling subnet is used to upsample and restore the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

[0046] For the detailed description of step S210 to step S220, please refer to step S110 to step S120, which will not be repeated here.

[0047] Step S230: Acquire an input image for executing the task to be processed.

[0048] In some embodiments, the input image for executing the task to be processed can be obtained through a trigger instruction in a device deployed with a visual network. The trigger instruction can be an instruction generated by an image to be processed uploaded by a user after triggering the execution of a visual task in the device. After the user triggers the execution of the visual task, the user uploads the image, and after uploading the image, the uploaded image is added to the trigger instruction, so that the input image for executing the task to be processed can be obtained in the trigger instruction. Optionally, the trigger instruction also includes the task type of the task to be processed.

[0049] Step S240: performing downsampling and block processing on the input image through the downsampling subnet to obtain a plurality of image blocks, and performing block fusion processing on the plurality of image blocks to obtain an intermediate image.

[0050] In some embodiments, in order to enable the visual network to quickly perceive image information in an image when performing a preset visual task, and to avoid large memory usage caused by excessive image information in the process of perceiving image information, the input image can be first downsampled through a downsampling subnet to reduce the resolution of the input image, thereby increasing the receptive field of the input image.

[0051] Optionally, after downsampling, in order to quickly perceive the image information of the input image, the downsampling subnet may divide the input image into blocks to obtain smaller image blocks, and then the downsampling subnet may perceive the image information in the smaller image blocks respectively, so as to obtain the features corresponding to each image block. Figure 3 As shown in the figure, the downsampling subnet divides the input image into four image blocks and perceives the image information based on the receptive fields corresponding to each of the four image blocks.

[0052] Optionally, in order to facilitate the subsequent feature space conversion subnet to process the image output by the downsampling subnet, the downsampling subnet can be used to perform block fusion processing on the divided image blocks, so that the downsampling subnet can output a complete image, so that the feature space conversion subnet can process based on the output image including global information. Figure 3 As shown, after the downsampling subnet perceives the image information based on the receptive fields corresponding to the four image blocks, the four image blocks are block fused to obtain an intermediate image, which includes the image information perceived by each image block based on its corresponding receptive field.

[0053] In some embodiments, step S240 includes:

[0054] An overlapping area between every two adjacent image blocks in the plurality of image blocks is determined.

[0055] In some embodiments, the feature dimension conversion subnet performs multiple convolutions on the intermediate image after block fusion for subsequent processing. However, after the input image is directly divided into blocks and then fused, since the distribution of each image block is not completely consistent, there is an obvious dividing line between the fused image blocks. The distribution difference between the image blocks after multiple convolutions is amplified. If it is directly fused, the visual dividing line on each image block will have a sudden change, which will lead to obvious dividing line artifacts at the edges of the input image divided in the target feature image obtained after the visual network performs a preset visual task. In order to avoid this influence, this embodiment determines the overlapping area between each two adjacent image blocks in the multiple image blocks. The overlapping area can be understood as the overlapping part between each two adjacent image blocks.

[0056] Optionally, the size of the overlapping area may be preset, and based on the size of the overlapping area, the overlapping area between each two adjacent image blocks in the plurality of image blocks is determined based on the size of the overlapping area. Optionally, when dividing the input image into blocks, multiple overlapping areas may be determined in the input image based on the preset size of the overlapping area, and the input image may be divided into blocks based on the overlapping areas, wherein the overlapping areas may be evenly distributed at adjacent locations of two adjacent image blocks, that is, the overlapping areas are evenly divided by the adjacent image blocks.

[0057] The multiple image blocks are fused based on the overlapping area to obtain the intermediate image.

[0058] In some embodiments, since the overlapping area can be regarded as the overlapping portion of two adjacent image blocks, each two adjacent image blocks in the multiple image blocks can be fused through the overlapping area, thereby achieving fusion processing of the multiple image blocks to obtain an intermediate image.

[0059] Optionally, since directly performing block fusion based on the overlapping area may cause the size of the fused image to be different from the size of the input image, in order to avoid this situation, the fusion weight of the overlapping area between adjacent image blocks can be first determined, and then the multiple image blocks can be fused based on the fusion weight and the overlapping area.

[0060] In some embodiments, the fusing of the multiple image blocks based on the overlapping area to obtain the intermediate image also includes: determining a fusion weight between each two adjacent image blocks based on the pixel values ​​of the overlapping area between each two adjacent image blocks; and fusing each two adjacent image blocks among the multiple image blocks based on the fusion weight to obtain the intermediate image.

[0061] In some embodiments, the pixel values ​​corresponding to the corner points of the overlapping area between each two adjacent image blocks and the pixel values ​​corresponding to the corner points of the two adjacent image blocks can be determined first, and then the distance between the corner points of the overlapping area and the corner points of the adjacent image blocks can be determined based on the pixel values ​​corresponding to the corner points of the overlapping area and the pixel values ​​corresponding to the corner points of the two adjacent image blocks, and then the fusion weight between each adjacent image block can be determined based on the distance. Optionally, the distance can be the Euclidean distance, the shortest distance or the cosine distance between the corner points of the overlapping area and the corner points of the adjacent image blocks, and the specific distance calculation method can be set according to actual needs.

[0062] Optionally, after determining the fusion weight between each two adjacent image blocks, the proportion of the overlapping area between the adjacent image blocks can be determined based on the fusion weight, and the overlapping area and the adjacent image blocks can be located based on the distance between each corner point of the overlapping area indicated by the fusion weight and each corner point of the adjacent image block. Then, the adjacent image blocks are block-fused based on the proportion of the overlapping area between the adjacent image blocks and the overlapping area, so as to obtain an intermediate image.

[0063] In some embodiments, the step S240 further includes:

[0064] Determine a first functional relationship and a second functional relationship, wherein the first functional relationship is used to characterize the relationship between the block size of the input image, the area size of the overlapping area, the number of blocks of multiple image blocks, and the image size of the input image, and the second functional relationship is used to characterize the relationship between the redundancy rate, the block size of the input image, the number of blocks of multiple image blocks, and the image size of the input image, and the redundancy rate is used to characterize the relationship between the block size of the input image, the number of blocks of multiple image blocks, and the image size of the input image.

[0065] In some embodiments, when the input image is processed in blocks, the size of the image blocks and the size of the overlapping area are parameters of the block processing. If the size of the overlapping area is too small, there will be obvious boundaries between adjacent image blocks in the intermediate image obtained after the block fusion processing. If the size of the overlapping area is too large, it will cause redundant calculations; if the size of the image blocks is too small, it will cause a waste of computing resources of the device; if the size of the image blocks is too large, it will cause memory swapping, which will lead to a decrease in the efficiency of performing the preset visual task. Therefore, the setting of the block size of the image blocks to be segmented and the area size of the overlapping area is crucial to the reasoning performance of the visual network in performing the preset visual task.

[0066] Optionally, the block size of the input image and the area size of the overlapping area can be evaluated by calculating the redundancy rate of the block size of the input image and the number of blocks of the multiple image blocks and the image size of the input image, wherein the image size of the input image is related to the block size of the input image, the area size of the overlapping area, and the number of blocks of the multiple image blocks. Therefore, the first functional relationship can be determined by the relationship between the image size of the input image and the block size of the input image, the area size of the overlapping area, and the number of blocks of the multiple image blocks, and the second functional relationship can be determined based on the redundancy rate of the block size of the input image and the number of blocks of the multiple image blocks and the image size of the input image. The first functional relationship can be PS*PN-OLS*(PN-1)≥IS, wherein IS is the image size of the input image, PS is the block size of the input image, PN is the number of blocks of the multiple image blocks, and OLS is the area size of the overlapping area; the second functional relationship can be Rrate=(PS*PN) / IS, wherein Rrate is the redundancy rate.

[0067] A current image size of the input image is determined, and a current number of blocks of the plurality of image blocks is determined based on the number of downsampling times.

[0068] In some embodiments, in order to ensure that the size of the overlapping area can include the receptive field limit of a pixel, half of the image size can be used as the maximum value and traversed in steps of 2 according to the empirical values ​​of multiple experiments, so as to perform block division, wherein the number of blocks of multiple image blocks can be set to 2. n , where n is the number of downsampling times performed by the downsampling subnet. After many experiments, it is found that when the number of downsampling times is 3, the efficiency and effect of reasoning on the image blocks obtained by the visual network when the number of downsampling times is 3 are the best. Therefore, the input image can be processed in blocks when the number of downsampling times is 3. Optionally, the input image can be directly framed to determine the current image size of the input image, or the input image size can be determined by analyzing the input image.

[0069] In some embodiments, the downsampling subnet performs block processing on the input image when the number of downsampling processes is 3 times.

[0070] In some embodiments, when the downsampling subnet performs block processing on the input image, the timing of segmenting the input image becomes a crucial issue. In order to enable as many layers as possible to be processed in blocks, the more layers that are processed in blocks, the larger the size of the overlapping area will be set, which will lead to more redundant calculations. Therefore, after many experiments, it is determined that the best effect is to segment the input image when downsampling 3 times. Therefore, the number of times the downsampling subnet downsamples the input image is set to 3 times, and the input image is segmented when the sampling subnet downsamples 3 times.

[0071] The redundancy rate is solved based on the current image size, the current number of blocks, the first functional relationship and the second functional relationship to obtain a parameter combination for the minimum redundancy rate, wherein the parameter combination includes the block size of the input image and the area size of the overlapping area.

[0072] In some embodiments, after determining the current image size and the current number of blocks of the input image, the current image size and the current number of blocks are substituted into the first functional relationship and the second functional relationship respectively, and the redundancy rate is solved by greedy search, so as to obtain a parameter combination of the optimal solution including the block size of the input image and the area size of the overlapping area. Optionally, in order to quickly determine the optimal solution, the block size can be set to be able to divide the number of downsampling times.

[0073] The input image is processed into blocks according to the parameter combination to obtain the multiple image blocks.

[0074] In some embodiments, after the parameter combination is determined, a dividing line for dividing the input image into blocks can be determined in the input image based on the determined block size of the input image, and then the overlapping area between each adjacent image block is determined based on the position of the dividing line and the area size of the overlapping area, so as to perform block processing on the input image based on the overlapping area and the dividing line. Optionally, when performing block processing on the input image, the input image can be uniformly divided into blocks based on the block size of the input image and the area size of the overlapping area, so as to ensure that each image block can include a receptive field of a pixel.

[0075] Please continue reading Figure 2 , step S250: performing feature extraction processing and feature dimension conversion processing on the intermediate image through the feature space conversion subnet to obtain an initial feature image.

[0076] In some embodiments, since the image information perceived by the downsampling subnet is the image information corresponding to each image block, in order to determine the global image information of the input image, the feature space conversion subnet can be used to extract features from the intermediate image after the block fusion of the downsampling subnet, wherein the intermediate image includes the corresponding image information obtained after perceiving each image block, and the feature space conversion subnet is used to extract features from the intermediate image to obtain the initial features corresponding to the intermediate image.

[0077] Optionally, since the visual network maps low-dimensional features to high-dimensional features when performing visual tasks, it can perform corresponding visual tasks based on high-dimensional features, thereby improving the accuracy of the visual network in performing visual tasks. Therefore, in an embodiment of the present application, the initial features obtained after feature extraction can be processed by feature space conversion subnet to perform feature dimension conversion, so as to obtain high-dimensional features and thus obtain the initial feature image. Figure 4 As shown, the feature space conversion subnet extracts features from the intermediate image output by the downsampling subnet, and then performs feature dimension conversion on the extracted features to obtain an initial feature image with high-dimensional features.

[0078] Step S260: upsampling and restoring the initial feature image through the upsampling subnet to obtain a target feature image, and executing the task to be processed on the target feature image.

[0079] In some embodiments, since the initial feature image output by the feature space conversion subnet is obtained by processing the image after the downsampling subnet downsamples the input image, and the downsampling of the input image will cause the resolution of the input image to change, and the visual network cannot lose the image quality when performing the preset visual task, it is necessary to upsample the initial feature map to restore the resolution of the initial feature map to obtain the target feature image, the size and resolution of the target feature image are the same as the size and resolution of the input image.

[0080] Optionally, the preset visual tasks may include, but are not limited to, image restoration, image enhancement, image feature extraction, image denoising, and image style transfer, so that after the upsampling subnet performs upsampling restoration, it can process the target feature image based on the preset visual task corresponding to the specific task to be processed, so as to output the target image after the task to be processed is performed. For example, if the preset visual task is specifically image denoising, the target feature image may be a feature map indicating the noise corresponding to the input image, so that the features corresponding to the noise in the target feature map are deleted, so as to obtain the denoised target image.

[0081] In this embodiment, after obtaining the input image for executing the task to be processed, the input image is first downsampled, divided into blocks and fused based on the downsampling subnet, and then the feature space conversion subnet performs feature extraction and feature dimension conversion on the output image of the downsampling subnet. Finally, the upsampling subnet performs upsampling and restoration on the output image of the feature space conversion subnet, and executes the task to be processed on the restored image, thereby reducing multiple memory exchange operations caused by insufficient memory when moving large blocks of data, and improving the processing efficiency of the visual network in executing preset visual tasks.

[0082] See also Figure 5 , Figure 5 The flowchart of the processing method of the visual network provided by an embodiment of the present application is shown in FIG. Figure 5 The process shown is described in detail, and the processing method of the visual network may specifically include the following steps:

[0083] Step S310: During deployment of the visual network in a device, determine the number of convolution channels corresponding to the device.

[0084] In some implementations, when the same visual network is deployed in different devices, the parameters of the visual network need to be adapted to the device to ensure that the corresponding visual tasks can be performed through the visual network in the device, and the chip of the device is usually inferred according to a specific channel alignment method, for example, some chips are based on 32 channels and some chips are based on 16 channels, so for different device-side deployment conditions, the structure of the network must also be fine-tuned to adapt to the corresponding hardware. Therefore, in order to ensure that the visual network in the device can work normally, the number of convolution channels in the device can be determined first, so as to adapt the visual network according to the number of convolution channels in the device.

[0085] Optionally, the number of convolution channels can be determined by obtaining a channel identifier in the device, where the channel identifier is used to indicate the number of convolution channels corresponding to the device. The channel identifier is set before the device leaves the factory or when the system is installed. The channel identifier can be obtained in the device to determine the number of convolution channels corresponding to the device.

[0086] Step S320: If the number of convolution channels corresponding to the visual network is different from the number of convolution channels corresponding to the device, the number of convolution channels of the visual network is adjusted to the number of convolution channels corresponding to the device.

[0087] In some embodiments, in a visual network, different numbers of convolution channels will result in different time lengths required to execute corresponding visual tasks. In order to ensure that the device side can better adapt to the visual network, the number of convolution channels corresponding to the visual network can be adjusted based on the number of convolution channels corresponding to the device.

[0088] Optionally, when it is determined that the number of convolution channels corresponding to the visual network is different from the number of convolution channels corresponding to the device, the device may not be able to support the visual network to perform visual tasks. Therefore, the number of convolution channels of the visual network needs to be adjusted. In order to adapt the visual network to the device, the number of convolution channels corresponding to the visual network can be directly adjusted to the number of convolution channels corresponding to the device, so that the number of convolution channels corresponding to the visual network is consistent with the number of convolution channels corresponding to the device, thereby ensuring that the chip of the device can support the visual network to perform visual tasks.

[0089] Optionally, the size relationship between the number of convolution channels corresponding to the device and the number of convolution channels of the visual network can be determined first, and then the adjustment strategy for the number of convolution channels corresponding to the visual network can be determined based on the size relationship. Optionally, if the size relationship indicates that the number of convolution channels corresponding to the device is greater than or equal to the number of convolution channels of the visual network, it can be determined that the chip of the device can support the visual network to perform visual tasks, and it is determined that there is no need to adjust the number of convolution channels of the visual network; if the size relationship indicates that the number of convolution channels corresponding to the device is less than the number of convolution channels of the visual network, it can be determined that the chip of the device cannot support the visual network to perform visual tasks, and therefore, the number of convolution channels corresponding to the visual network can be adjusted to the number of convolution channels corresponding to the device.

[0090] Since the time required to execute visual tasks corresponding to different numbers of convolution channels used by the visual network is different, as shown in Table 1 below, we can first determine whether the difference between the number of convolution channels corresponding to the device and the number of convolution channels of the visual network is small, so as to determine whether to adjust the number of convolution channels of the visual network to the number of convolution channels corresponding to the device.

[0091] Table 1 Execution time corresponding to different numbers of convolution channels

[0092]

[0093] Among them, for devices using model A chips, the corresponding delay times when the number of convolution channels is 32, 33, and 64 respectively, the corresponding delay time when the number of convolution channels is 64 (model A chip (execute) time-Accelerator (execute) time) is 11.2ms, the corresponding delay time when the number of convolution channels is 33 is 5.9ms, and the corresponding delay time when the number of convolution channels is 32 is 5.6ms, that is, the difference between the number of convolution channels of 33 and the number of convolution channels of 32 is not large, but the delay time when the number of convolution channels is 64 is the longest. Therefore, when the difference between the number of convolution channels corresponding to the device and the number of convolution channels of the visual network is small, the number of convolution channels of the visual network can be directly adjusted to the number of convolution channels corresponding to the device, so that the visual network and the device are more compatible.

[0094] Step S330: Determine the task type of the task to be processed in the visual network.

[0095] Step S340: If the task type is a preset visual task type, the visual network is split into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, wherein: the downsampling subnet is used to downsample, block and fusion the input image; the feature space conversion subnet is used to perform feature extraction and feature dimension conversion on the output image of the downsampling subnet; the upsampling subnet is used to upsample and restore the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

[0096] For the detailed description of step S330 to step S340, please refer to step S110 to step S120, which will not be repeated here.

[0097] In this embodiment, during the process of deploying a visual network on a device, the number of convolution channels corresponding to the device is determined. When it is determined that the number of convolution channels of the device is different from the number of convolution channels of the visual network, the number of convolution channels of the visual network is adjusted to the number of convolution channels of the device. In this way, the number of convolution channels of the visual network is the same as the number of convolution channels of the device, making the visual network more compatible with the device and improving the efficiency of the visual network in executing data tasks in the device.

[0098] See also Figure 6 , Figure 6 The flowchart of the processing method of the visual network provided by an embodiment of the present application is shown in FIG. Figure 6 The process shown is described in detail, and the processing method of the visual network may specifically include the following steps:

[0099] Step S410: Determine a plurality of subgraphs in the visual network.

[0100] In some embodiments, the visual network usually designs various long jump connections, short jump connections and attention structures, and different subgraphs can be obtained by designing a combination of multiple operators, so that the subgraphs can achieve different structural designs. However, since the design of the structure of the visual network may result in low utilization of the device's processor, the utilization of the device's processor can be improved by adjusting the subgraphs in the visual network, that is, optimizing the subgraphs of the visual network. Optionally, multiple subgraphs can be determined by operators with associated relationships in the visual network, where the associated relationship is used to indicate the input-output relationship between multiple operators.

[0101] Step S420: If there is a target subgraph including a global operator, a convolution operator and a multiplication operator in the multiple subgraphs, then the target subgraph is deleted in the visual network.

[0102] In some embodiments, the visual network can perform global perception, convolution calculation, and multiplication calculation on the input image in sequence to achieve self-attention operation to capture information from multiple angles, such as Figure 7 As shown, the target subgraph selected by the frame first performs global information interaction based on the global operator, then performs convolution calculation based on the convolution operator, and finally performs calculation based on the multiplication operator. However, when the visual network performs the preset visual task, the calculation of the target subgraph does not have a significant impact on the execution of the preset visual task, but it will cause excessive memory swaps, thereby resulting in low processor utilization of the device. And due to the global operator, it will be difficult to perform block processing. Therefore, if it is determined that there is a target subgraph including a global operator, a convolution operator, and a multiplication operator in multiple subgraphs, the target subgraph is deleted in the visual network. When there is a target subgraph in the visual network, the device has a large amount of direct memory access (DMA) waiting time when performing data transmission, where DMA waiting generally refers to a waiting state caused by certain conditions not being met during DMA transmission. As shown Figure 8 As shown, the blank space in the box is the DMA waiting time corresponding to the device when the visual network including the target operator performs the preset visual task.

[0103] When there is no target subgraph in the visual network, there is no need to execute global operators, convolution operators, and multiplication operators, thereby reducing the DMA waiting time of the device, such as Fig. 9 The blank space in the box shown is the DMA waiting time corresponding to the device when the visual network including the target operator performs the preset visual task. It can be intuitively seen that the DMA waiting time distribution of the device changes from sparse to dense.

[0104] Step S430: Determine the task type of the task to be processed in the visual network.

[0105] Step S440: If the task type is a preset visual task type, the visual network is split into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, wherein: the downsampling subnet is used to perform downsampling, block processing and block fusion processing on the input image; the feature space conversion subnet is used to perform feature extraction and feature dimension conversion processing on the output image of the downsampling subnet; the upsampling subnet is used to perform upsampling and restoration processing on the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

[0106] For the detailed description of step S430 to step S440, please refer to step S110 to step S120, which will not be repeated here.

[0107] In this embodiment, the target subgraph including the global operator, the convolution operator and the multiplication operator in the visual network can be deleted, so as to achieve subgraph optimization of the visual network, thereby improving the utilization rate of the processor of the device.

[0108] See also Fig.10 , Fig.10 The flowchart of the processing method of the visual network provided by an embodiment of the present application is shown in FIG. Fig.10 The process shown is described in detail, and the processing method of the visual network may specifically include the following steps:

[0109] Step S510: During deployment of the visual network in a device, a target operator in the visual network is determined, wherein the target operator is related to the deployment of the visual network.

[0110] In some embodiments, when deploying a visual network in a device, the deployment can be performed using a commonly used deployment tool, such as a model A chip deployment tool or a neural processing engine deployment tool. However, when deploying a visual network, these deployment tools may generate some slicing operations during the compilation of the network structure, such as slice operations. Fig.11 As shown, these operations are only related to the deployment of the visual network in the device and do not involve the subsequent processing of the input image. The corresponding target operator related to the deployment of the visual network can be determined in the network result after deployment compilation through the corresponding operation identifier of the deployment operation.

[0111] Step S520: deleting the target operator in the visual network.

[0112] In some embodiments, since the deployment-related operations corresponding to the target operator have no effect when the visual network subsequently executes the visual network, in order to improve the efficiency of the visual network in executing preset visual tasks, the target operator can be deleted, so that there is no need to execute the target operator when the visual network executes data tasks.

[0113] Step S530: Determine the task type of the task to be processed in the visual network.

[0114] Step S540: If the task type is a preset visual task type, the visual network is split into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, wherein: the downsampling subnet is used to perform downsampling, block processing and block fusion processing on the input image; the feature space conversion subnet is used to perform feature extraction and feature dimension conversion processing on the output image of the downsampling subnet; the upsampling subnet is used to perform upsampling and restoration processing on the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

[0115] For the detailed description of step S530 to step S540, please refer to step S110 to step S120, which will not be repeated here.

[0116] In this embodiment, during the process of deploying the visual network in the device, the deployment is achieved through a target operator related to the visual network, but the target operator is not used in the subsequent process of processing the input image, that is, the deployment-related operations corresponding to the target operator have no effect when the subsequent visual network executes the preset visual network. In this way, the processing efficiency of the visual network when executing the preset visual tasks is improved by deleting the target operator.

[0117] See also Fig.12 , Fig.12 The flowchart of the processing method of the visual network provided by an embodiment of the present application is shown in FIG. Fig.12 The process shown is described in detail, and the processing method of the visual network may specifically include the following steps:

[0118] Step S610: Determine a combination operator in the visual network, wherein the combination operator is used to indicate that a convolution value is obtained after executing the convolution operator and then stored, and then a multiplication operator is executed based on the stored convolution value.

[0119] In some implementations, there is usually a combination operator of convolution and multiplication in the visual network to realize the functions of feature extraction, information interaction and model compression. However, during the execution of the convolution operator and the multiplication operator in the combination operator, the convolution operator will be executed first to obtain the convolution value, and in order to facilitate the subsequent multiplication operator to be executed, the convolution value needs to be stored first, and then the convolution value is read from the memory when the multiplication operator is executed, such as Fig.13 Optionally, the combination operator can be determined by determining the output of the convolution operator in the visual network as the adjacent convolution operator and the multiplication operator of the input of the multiplication operator.

[0120] Step S620: determining a bias weight based on the multiplication operator in the combination operator, and determining a target operator based on the bias weight and the convolution operator, wherein the target operator is used to perform convolution calculation on the input.

[0121] In some implementations, since the combination operator needs to be executed step by step, the read and write time of the input and output is increased, resulting in low processing efficiency of the visual network. Therefore, the convolution operator and the multiplication operator can be used as one operator to reduce the read and write time of the input and output and improve the processing efficiency of the visual network.

[0122] Optionally, the bias weight in the multiplication operator can be determined first, and then the bias weight can be set in the convolution operator, so that only the convolution operator calculation is performed in the device's processor, reducing the calculation of the multiplication operator, that is, the visual network only needs to perform one convolution when performing a visual task. Optionally, the target operator can be determined by the formula Mul(Conv(x,w,b1),b2)=Conv(x,wb2,b1b2), where Conv(x,w,b1) is the convolution operator in the combination operator, Mul(Conv(x,w,b1),b2) is the multiplication operator performed based on the stored convolution value, x is the input, w is the weight tensor of the convolution kernel, b1 is the bias weight corresponding to the convolution kernel, b2 is the bias weight of the multiplication operator, and in the target operator Conv(x,wb2,b1b2), b1b2 is the bias weight determined based on the multiplication operator, and wb2 is the convolution kernel that combines the convolution operator and the target operator.

[0123] Step S630: Replace the combination operator with the target operator.

[0124] In some embodiments, the combination operator is replaced with the target operator, and its corresponding input and output do not change, but only one convolution is performed, which reduces the operations of storing convolution values ​​and reading convolution values ​​in the combination operator when executing the multiplication operator, thereby improving the processing efficiency of the visual network.

[0125] Step S640: Determine the task type of the task to be processed in the visual network.

[0126] Step S650: If the task type is a preset visual task type, the visual network is split into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, wherein: the downsampling subnet is used to perform downsampling, block processing and block fusion processing on the input image; the feature space conversion subnet is used to perform feature extraction and feature dimension conversion processing on the output image of the downsampling subnet; the upsampling subnet is used to perform upsampling and restoration processing on the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

[0127] For the detailed description of step S640 to step S650, please refer to step S110 to step S120, which will not be repeated here.

[0128] In some embodiments, the downsampling subnetwork downsamples the input image by nearest neighbor interpolation.

[0129] In some embodiments, the downsampling method used by the visual network when reducing the resolution is usually designed to directly use bilinear interpolation. However, when performing the downsampling process, the visual network is not sensitive to the selected interpolation method. The use of bilinear interpolation may increase the number of multiplication and addition operations and discontinuous access operations, thereby causing the visual network to execute the preset visual network inefficiently. Therefore, the input image can be downsampled by using the nearest neighbor interpolation method, thereby reducing the number of multiplication and addition operations, reducing discontinuous access, and increasing the memory hit rate, thereby reducing time consumption and improving the processing efficiency of the visual network.

[0130] In some embodiments, the visual network performs convolution processing by zero padding.

[0131] In some implementations, when performing convolution processing on a visual network, there is usually no specific convolution method specified. Different convolution methods have different corresponding time consumptions. For example, when performing a preset visual task, the reflection filling method may have the problem of applying for additional memory and consuming a lot of time, which may increase the waiting time of the HMX of the hardware module dedicated to deep learning calculations. However, the zero filling method does not have these effects. Fig.14 As shown in Figure 2, when zero padding is used for convolution processing, the waiting time of HMX is very small, while Fig.15 As shown in the figure, when using the reflection padding method for convolution processing, there is a lot of HMX waiting time. Therefore, the visual network can be set to perform convolution processing by zero padding.

[0132] In this embodiment, the bias weight is determined by combining the multiplication operator in the combination operator determined in the visual network, and then the target operator is determined based on the convolution operator and the bias weight in the combination operator determined in the visual network, and then the combination operator in the visual network is replaced with the target operator, so as to reduce the reading and writing time of the visual network when executing the combination operator when performing a preset visual task, thereby improving the processing efficiency of the visual network.

[0133] In summary, the processing method, device, electronic device and storage medium of the visual network provided in the embodiments of the present application determine whether to split the visual network according to the task type of the task to be processed of the determined visual network, and when the task type is a preset visual task type, split the visual network into a downsampling subnet, a feature space conversion subnet and an upsampling subnet, so that the input image is downsampled, blocked and fused by the downsampling subnet, and then the feature space conversion subnet performs feature extraction and feature dimension conversion on the output image of the downsampling subnet, and finally the upsampling subnet performs upsampling and restoration on the output image of the feature space conversion subnet, and executes the task to be processed on the restored image, thereby reducing multiple memory exchange operations caused by insufficient memory when moving large blocks of data, improving the processing efficiency of the visual network in performing preset visual tasks, and optimizing the visual network from different levels of the global network structure, the intermediate layer subnet and the operator optimization, thereby greatly reducing the network inference delay, memory and memory access space, making the deployment of the visual network that performs visual tasks on the platform side more friendly.

[0134] See also Fig.16 , Fig.16 The module block diagram of the processing device of the visual network provided by an embodiment of the present application is shown. Fig.16 The block diagram shown in the figure is used to illustrate that the processing device 200 of the visual network includes: a task type determination module 210 and a splitting module 220, wherein:

[0135] A task type determination module 210, used to determine the task type of the task to be processed by the visual network;

[0136] The splitting module 220 is used to split the visual network to obtain a downsampling subnet, a feature space conversion subnet and an upsampling subnet if the task type is a preset visual task type, wherein:

[0137] The downsampling subnet is used to perform downsampling processing, block processing and block fusion processing on the input image;

[0138] The feature space conversion subnet is used to perform feature extraction processing and feature dimension conversion processing on the output image of the downsampling subnet;

[0139] The upsampling subnet is used to perform upsampling and restoration processing on the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

[0140] Furthermore, the visual network processing 200 also includes an input image acquisition module, an intermediate image determination module, an initial feature image determination module and a target feature image determination module, wherein:

[0141] An input image acquisition module, used to acquire an input image for executing the task to be processed;

[0142] An intermediate image determination module, configured to perform downsampling and block processing on the input image through the downsampling subnet to obtain a plurality of image blocks, and perform block fusion processing on the plurality of image blocks to obtain an intermediate image;

[0143] An initial feature image determination module, used to perform feature extraction processing and feature dimension conversion processing on the intermediate image through the feature space conversion subnet to obtain an initial feature image;

[0144] The target feature image determination module is used to upsample and restore the initial feature image through the upsampling subnet to obtain the target feature image, and execute the task to be processed on the target feature image.

[0145] Further, the intermediate image determination module includes an overlapping area determination submodule and an intermediate image determination submodule, wherein:

[0146] An overlapping region determining submodule, used to determine an overlapping region between every two adjacent image blocks in the plurality of image blocks;

[0147] The intermediate image determination submodule is used to perform fusion processing on the multiple image blocks based on the overlapping area to obtain the intermediate image.

[0148] Further, the intermediate image determination submodule includes a fusion weight determination unit and an intermediate image determination unit, wherein:

[0149] A fusion weight determination unit, used to determine the fusion weight between each two adjacent image blocks according to the pixel values ​​of the overlapping area between each two adjacent image blocks;

[0150] The intermediate image determining unit is used to perform fusion processing on every two adjacent image blocks among the multiple image blocks based on the fusion weight to obtain the intermediate image.

[0151] Furthermore, the intermediate image determination module includes a function relationship determination submodule, a current block quantity determination submodule, a parameter combination determination submodule and a block processing submodule, wherein:

[0152] a functional relationship determination submodule, configured to determine a first functional relationship and a second functional relationship, wherein the first functional relationship is used to characterize the relationship between the block size of the input image, the area size of the overlapping area, the number of blocks of the plurality of image blocks, and the image size of the input image, and the second functional relationship is used to characterize the relationship between the redundancy rate, the block size of the input image, the number of blocks of the plurality of image blocks, and the image size of the input image, and the redundancy rate is used to characterize the relationship between the block size of the input image, the number of blocks of the plurality of image blocks, and the image size of the input image;

[0153] a current block number determination submodule, configured to determine a current image size of the input image, and determine a current block number of the plurality of image blocks based on the number of downsampling times;

[0154] a parameter combination determination submodule, configured to solve the redundancy rate based on the current image size, the current number of blocks, the first functional relationship, and the second functional relationship to obtain a parameter combination with a minimum redundancy rate, wherein the parameter combination includes a block size of the input image and an area size of the overlapping area;

[0155] The block processing submodule is used to perform block processing on the input image according to the parameter combination to obtain the multiple image blocks.

[0156] Furthermore, the downsampling subnet performs block processing on the input image when the number of downsampling processing is 3 times.

[0157] Furthermore, the visual network processing 200 also includes a convolution channel number determination submodule and a channel number adjustment submodule, wherein:

[0158] A convolution channel number determination submodule, used to determine the number of convolution channels corresponding to the device during the process of deploying the visual network in the device;

[0159] The channel number adjustment submodule is used to adjust the number of convolution channels of the visual network to the number of convolution channels corresponding to the device if the number of convolution channels corresponding to the visual network is different from the number of convolution channels corresponding to the device.

[0160] Furthermore, the visual network processing 200 also includes a sub-graph determination sub-module and a target sub-graph processing sub-module, wherein:

[0161] A subgraph determination submodule, used to determine a plurality of subgraphs in the visual network;

[0162] The target subgraph processing submodule is used to delete the target subgraph in the visual network if there is a target subgraph including a global operator, a convolution operator and a multiplication operator in the multiple subgraphs.

[0163] Furthermore, the downsampling subnetwork performs downsampling processing on the input image by means of nearest neighbor interpolation.

[0164] Furthermore, the visual network performs convolution processing by means of zero padding.

[0165] Furthermore, the visual network processing 200 also includes a target operator determination submodule and a target operator processing submodule, wherein:

[0166] A target operator determination submodule, used to determine a target operator in the visual network during deployment of the visual network in a device, wherein the target operator is related to the deployment of the visual network;

[0167] The target operator processing submodule is used to delete the target operator in the visual network.

[0168] Furthermore, the visual network processing 200 also includes a combination operator determination submodule, a target operator determination submodule and a replacement submodule, wherein:

[0169] A combination operator determination submodule, used to determine a combination operator in the visual network, wherein the combination operator is used to indicate that a convolution value is obtained after executing the convolution operator and then a multiplication operator is executed based on the stored convolution value;

[0170] A target operator determination submodule, used to determine a bias weight based on a multiplication operator in the combination operator, and to determine a target operator based on the bias weight and the convolution operator, wherein the target operator is used to perform convolution calculation on an input;

[0171] The replacement submodule is used to replace the combination operator with the target operator.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0173] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0174] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0175] See also Fig.17, which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0176] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0177] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0178] See also Fig.18, which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0179] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 300 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing a visual network, characterized in that: The method comprises: Determine the task type of the task to be processed by the visual network; If the task type is a preset visual task type, the visual network is split to obtain a downsampling subnet, a feature space conversion subnet and an upsampling subnet, wherein: The downsampling subnet is used to perform downsampling processing, block processing and block fusion processing on the input image; The feature space conversion subnet is used to perform feature extraction processing and feature dimension conversion processing on the output image of the downsampling subnet; The upsampling subnet is used to perform upsampling and restoration processing on the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

2. The method according to claim 1, characterized in that If the task type is a preset visual task type, the visual network is split to obtain a downsampling subnet, a feature space conversion subnet and an upsampling subnet, further comprising: Acquire an input image for executing the task to be processed; Performing downsampling and block processing on the input image through the downsampling subnet to obtain a plurality of image blocks, and performing block fusion processing on the plurality of image blocks to obtain an intermediate image; Performing feature extraction and feature dimension conversion processing on the intermediate image through the feature space conversion subnet to obtain an initial feature image; The initial feature image is upsampled and restored by the upsampling subnet to obtain a target feature image, and the task to be processed is performed on the target feature image.

3. The method according to claim 2, characterized in that The performing block fusion processing on the multiple image blocks to obtain an intermediate image includes: Determine an overlapping area between every two adjacent image blocks in the plurality of image blocks; The multiple image blocks are fused based on the overlapping area to obtain the intermediate image.

4. The method according to claim 3, characterized in that The fusing the multiple image blocks based on the overlapping area to obtain the intermediate image includes: Determine the fusion weight between each two adjacent image blocks according to the pixel values ​​of the overlapping area between each two adjacent image blocks; Each two adjacent image blocks in the plurality of image blocks are fused based on the fusion weight to obtain the intermediate image.

5. The method according to claim 3, characterized in that The step of performing block processing on the input image by the downsampling subnet to obtain a plurality of image blocks includes: Determining a first functional relationship and a second functional relationship, wherein the first functional relationship is used to characterize the relationship between the block size of the input image, the area size of the overlapping area, the number of blocks of the plurality of image blocks, and the image size of the input image, and the second functional relationship is used to characterize the relationship between the redundancy rate, the block size of the input image, the number of blocks of the plurality of image blocks, and the image size of the input image, and the redundancy rate is used to characterize the relationship between the block size of the input image, the number of blocks of the plurality of image blocks, and the image size of the input image; Determining a current image size of the input image, and determining a current number of blocks of the plurality of image blocks based on the number of downsampling times; Solving the redundancy rate based on the current image size, the current number of blocks, the first functional relationship, and the second functional relationship to obtain a parameter combination of a minimum redundancy rate, wherein the parameter combination includes a block size of the input image and an area size of the overlapping area; The input image is processed into blocks according to the parameter combination to obtain the multiple image blocks.

6. The method according to any one of claims 1 to 5, characterized in that: The downsampling subnet performs block processing on the input image when the number of downsampling processing is 3 times.

7. The method according to any one of claims 1 to 5, characterized in that: Before determining the task type of the task to be processed by the visual network, the method further includes: In the process of deploying the visual network in a device, determining the number of convolution channels corresponding to the device; If the number of convolution channels corresponding to the visual network is different from the number of convolution channels corresponding to the device, the number of convolution channels of the visual network is adjusted to the number of convolution channels corresponding to the device.

8. The method according to any one of claims 1 to 5, characterized in that: Before determining the task type of the task to be processed by the visual network, the method further includes: determining a plurality of subgraphs in the visual network; If there is a target subgraph including a global operator, a convolution operator and a multiplication operator in the multiple subgraphs, the target subgraph is deleted in the visual network.

9. The method according to any one of claims 1 to 5, characterized in that: The downsampling subnetwork performs downsampling processing on the input image by means of nearest neighbor interpolation.

10. The method according to any one of claims 1 to 5, characterized in that: The visual network performs convolution processing by means of zero padding.

11. The method according to any one of claims 1 to 5, characterized in that: Before determining the task type of the task to be processed by the visual network, the method further includes: During deployment of the visual network in a device, determining a target operator in the visual network, wherein the target operator is related to deployment of the visual network; The target operator is deleted in the visual network.

12. The method according to any one of claims 1 to 5, characterized in that: Before determining the task type of the task to be processed by the visual network, the method further includes: Determine a combination operator in the visual network, wherein the combination operator is used to indicate that a convolution value is obtained after executing the convolution operator and then stored, and then a multiplication operator is executed based on the stored convolution value; Determining a bias weight based on a multiplication operator in the combination operator, and determining a target operator based on the bias weight and the convolution operator, wherein the target operator is used to perform convolution calculation on an input; The combination operator is replaced by the target operator.

13. A visual network processing device, characterized in that: The device comprises: A task type determination module, used to determine the task type of the task to be processed by the visual network; A splitting module is used for splitting the visual network to obtain a downsampling subnet, a feature space conversion subnet and an upsampling subnet if the task type is a preset visual task type, wherein: The downsampling subnet is used to perform downsampling processing, block processing and block fusion processing on the input image; The feature space conversion subnet is used to perform feature extraction processing and feature dimension conversion processing on the output image of the downsampling subnet; The upsampling subnet is used to perform upsampling and restoration processing on the output image of the feature space conversion subnet, and execute the task to be processed on the restored image.

14. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 12.

Images