Image processing method, device, system and computer readable storage medium
By compressing the feature value for each alternative layer in the neural network and selecting the appropriate compressed data amount and bit depth, the problem of low computing efficiency of neural networks is solved, and the performance of machine vision tasks and the compression rate of video data is improved. The calculation efficiency and the effect of saving calculation load is achieved.
Patent Information
- Application Number
- CN202110339333.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-03-30
AI Technical Summary
Neural networks are incomputed in processing large amounts of data, and data compression may affect performance, making it difficult to meet the needs of machine vision tasks.
By compressing the feature values for each alternative layer in the neural network, selecting the appropriate compressed data amount and bit depth according to the preset evaluation indicators, and optimizing the compression method of the feature values to ensure the computing efficiency of the neural network under task requirements.
It improves the computing efficiency of neural networks and the compression rate of video data, meets the performance requirements of machine vision tasks, and saves computing load.
Smart Images

Figure CN115150622B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device, system and computer-readable storage medium. Background Art
[0002] The goal of the VCM (Video Coding for Machines) expert group is to define a compressed video or feature code stream extracted from video that can be used for a variety of machine tasks while ensuring high compression efficiency and machine intelligence task performance, serving machine vision and human-machine hybrid vision applications.
[0003] Deep learning feature coding for machine vision can first determine the layer of the neural network where the coding object is located, output the feature coding through the determined layer of the neural network, and then use the feature coding for various subsequent video processing tasks. Summary of the invention
[0004] The inventors found that neural networks often include many layers and need to process a large amount of data. If the data in the neural network can be properly compressed, the computing efficiency can be improved. However, data compression cannot excessively reduce the performance of the neural network and needs to meet the task requirements. For deep learning feature coding for machine vision, it is necessary to first meet the machine vision task and then improve the efficiency of feature coding.
[0005] A technical problem to be solved by the present disclosure is: how to improve the computational efficiency of neural networks and meet task requirements.
[0006] According to some embodiments of the present disclosure, an image processing method is provided, including: for each alternative layer in a plurality of alternative layers in a neural network, compressing the characteristic value of an image output by the alternative layer in a preset manner to obtain a compressed characteristic value; inputting the compressed characteristic value into a layer connected after the alternative layer to determine an evaluation index value of the neural network; when the evaluation index value of the neural network reaches a preset value, determining the data volume of the compressed characteristic value corresponding to the alternative layer as a reference index of the alternative layer; selecting an alternative layer as a layer to be compressed according to the reference index of each alternative layer, so that the characteristic value output by the layer to be compressed can be compressed during the use of the neural network.
[0007] In some embodiments, compressing the characteristic values of the image output by the alternative layer in a preset manner to obtain compressed characteristic values includes: compressing the characteristic values of the image output by the alternative layer according to a current bit depth value to obtain compressed characteristic values, wherein the current bit depth value is greater than or equal to a preset minimum value; after determining the evaluation index value of the neural network, the method further includes: when the evaluation index value of the neural network does not reach the preset value, increasing the current bit depth value by a preset step size as the current bit depth value; and re-compressing the characteristic values of the image output by the alternative layer according to the current bit depth value until the current bit depth value reaches the preset maximum value, or the evaluation index value of the neural network reaches the preset value.
[0008] In some embodiments, determining the data amount of the compressed characteristic value corresponding to the alternative layer includes: summing or averaging the data amount of the compressed characteristic value of each image corresponding to the alternative layer as the data amount of the compressed characteristic value corresponding to the alternative layer.
[0009] In some embodiments, selecting a candidate layer as the layer to be compressed according to the reference indicators of the candidate layers includes: selecting a candidate layer with the smallest reference indicator as the layer to be compressed according to the reference indicators of the candidate layers.
[0010] In some embodiments, when the neural network is a feature pyramid network FPN, the candidate layers are selected from the layers before the layer that first enters the FPN lateral connection.
[0011] In some embodiments, the method further includes: inputting the image to be processed into the neural network; after the image to be processed passes through the layer to be compressed, compressing the feature values of the image to be processed output by the layer to be compressed according to a preset method to obtain compressed feature values; inputting the compressed feature values into the layer connected after the layer to be compressed to obtain the output result of the neural network.
[0012] In some embodiments, the layer to be compressed and the layers connected before the layer to be compressed in the neural network are deployed at the encoding end, and the layers connected after the layer to be compressed are deployed at the decoding end; inputting the compressed feature value into the layer connected after the layer to be compressed includes: the encoding end encodes the compressed feature value and sends it to the decoding end; the decoding end decodes and inputs the compressed feature value into the layer connected after the layer to be compressed.
[0013] In some embodiments, the encoding end is located at an edge computing node and the decoding end is located at the cloud.
[0014] In some embodiments, the evaluation index value of the neural network includes: accuracy, precision, recall rate or F value.
[0015] In some embodiments, according to the current bit depth value, the characteristic value of the image output by the alternative layer is compressed to obtain the compressed characteristic value, which includes: determining the current scale according to the characteristic value of the image output by the alternative layer and the current bit depth value; quantizing the characteristic value of the image output by the alternative layer according to the current scale; multiplying the quantized value by the scale to obtain the compressed characteristic value.
[0016] According to some other embodiments of the present disclosure, an image processing device is provided, including: a compression module, used for compressing the characteristic values of an image output by each alternative layer in a plurality of alternative layers in a neural network in a preset manner to obtain compressed characteristic values; an evaluation index determination module, used for inputting the compressed characteristic values into a layer connected after the alternative layer to determine the evaluation index value of the neural network; a reference index determination module, used for determining the data volume of the compressed characteristic values corresponding to the alternative layer as a reference index of the alternative layer when the evaluation index value of the neural network reaches a preset value; and a selection module, used for selecting an alternative layer as a layer to be compressed according to the reference indexes of each alternative layer, so as to compress the characteristic values output by the layer to be compressed during the use of the neural network.
[0017] In some embodiments, the method also includes: an image processing module, used to input the image to be processed into the neural network; after the image to be processed passes through the layer to be compressed, the feature value of the image to be processed output by the layer to be compressed is compressed according to a preset method to obtain a compressed feature value; the compressed feature value is input into the layer connected after the layer to be compressed to obtain the output result of the neural network.
[0018] According to some further embodiments of the present disclosure, an image processing device is provided, comprising: a processor; and a memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the image processing method as described in any of the aforementioned embodiments.
[0019] According to some further embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the image processing method of any of the aforementioned embodiments.
[0020] According to some further embodiments of the present disclosure, an image processing system is provided, including: an encoding end and a decoding end, wherein the encoding end includes the image processing device of any of the aforementioned embodiments, the layer to be compressed and the layers connected before the layer to be compressed in the neural network are deployed at the encoding end, and the layers connected after the layer to be compressed are deployed at the decoding end.
[0021] In the present disclosure, for each candidate layer in the neural network, the characteristic value of the image output by the candidate layer is compressed in a preset manner to obtain a compressed characteristic value. After the compressed characteristic value is processed in the subsequent layers of the neural network, an evaluation index value for evaluating the performance of the neural network can be obtained. If the evaluation index value of the neural network reaches the preset value, the data volume of the compressed characteristic value corresponding to the candidate layer is determined as the reference index of the candidate layer, and an candidate layer is selected as the layer to be compressed according to the reference index of each candidate layer. In the subsequent use of the neural network, the characteristic value of the image output by the layer to be compressed is compressed and then input into the subsequent layers of the neural network for processing. The method disclosed in the present disclosure can compress the data of the image processed by the neural network when the evaluation index of the neural network meets the task requirements, thereby improving the computational efficiency of the neural network. When the method disclosed in the present disclosure is applied to deep learning feature coding for machine vision, the compression rate and encoding efficiency of video data can be improved while meeting the machine vision task, and the computational load can be saved.
[0022] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0024] Figure 1 A schematic flowchart of an image processing method according to some embodiments of the present disclosure is shown.
[0025] Figure 2 A schematic flowchart showing image processing methods according to other embodiments of the present disclosure is shown.
[0026] Figure 3 A schematic flowchart showing image processing methods according to some further embodiments of the present disclosure is shown.
[0027] Figure 4 A schematic diagram showing the structural flow of an image processing device according to some embodiments of the present disclosure.
[0028] Figure 5 A schematic diagram showing the structural flow of an image processing device according to some other embodiments of the present disclosure.
[0029] Figure 6 A schematic diagram showing the structural flow of an image processing device according to some further embodiments of the present disclosure.
[0030] Figure 7 A schematic diagram showing the structural flow of an image processing system according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0031] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0032] The present disclosure proposes an image processing method, which is combined with Figures 1 to 3 Give a description.
[0033] Figure 1 Flowcharts of some embodiments of the image processing method disclosed herein. Figure 1 As shown, the method of this embodiment includes: steps S102 to S108.
[0034] In step S102, for each candidate layer in the plurality of candidate layers in the neural network, the characteristic value of the image output by the candidate layer is compressed according to a preset method to obtain a compressed characteristic value.
[0035] First, multiple images (for example, images in a video) can be input into a neural network, and the characteristic values of the outputs of each layer can be obtained after passing through each layer, and the characteristic values can be represented in the form of a feature tensor. The candidate layers can be multiple layers between the input layer and the output layer selected according to actual needs. In some embodiments, when the neural network is an FPN (feature pyramid network), the candidate layers are selected from the layers before the first layer that enters the FPN lateral connection.
[0036] The purpose of convolution operation in FPN is to extract different features of the input. The first convolution layer may only extract some low-level features such as edges, lines, and corners. More layers of the network can iteratively extract more complex features from low-level features. Considering the computational complexity of the encoding side and the fact that the FPN network will perform feature fusion at different layers, the FPN-based feature extraction layer for machine vision encoding can be located before the first layer that enters the FPN lateral connection in the bottom-up forward network.
[0037] In some embodiments, according to the current bit depth value (Bit Depth, BD), the characteristic value of the image output by the alternative layer is compressed to obtain a compressed characteristic value, and the current bit depth value is greater than or equal to the preset minimum value and less than or equal to the preset maximum value. For example, the value range of BD is an integer from 3 to 10, which is not limited to the examples given. The characteristic value of the image output by the alternative layer can be linearly quantized and dequantized according to the current BD value to obtain a compressed characteristic value. The compression method is not limited to linear quantization and dequantization, and can be selected according to actual needs, such as vector quantization, etc., which is not limited to the examples given. In subsequent embodiments, how to determine the current bit depth value and the process of dynamically adjusting the bit depth value will be described.
[0038] In some embodiments, the current quantization level is determined according to the characteristic value of the image output by the candidate layer and the current bit depth value; the characteristic value of the image output by the candidate layer is quantized according to the current quantization level; the quantized value is multiplied by the quantization level to obtain the compressed characteristic value. For example, the maximum value A and the minimum value B of the characteristic value of the image output by the candidate layer are determined, and the quantization level is determined to be M=2 BD , that is, the BD value is used as the exponent and 2 is the base. The order is Δ=(AB) / M. The eigenvalue of the image output by the candidate layer is divided by the order, and the obtained value is rounded down to obtain a quantized value, and the quantized value is multiplied by the order to obtain a compressed eigenvalue. Other methods can also be used to compress the eigenvalue, not limited to the examples given.
[0039] In step S104, the compressed feature value is input into the layer connected after the candidate layer to determine the evaluation index value of the neural network.
[0040] The compressed feature value can continue to be input into the layer connected after the candidate layer until the result is output by the output layer of the neural network, and the evaluation index value of the neural network can be determined according to the output result. In some embodiments, the evaluation index value of the neural network includes: accuracy, precision, recall or F value, not limited to the examples given. For example, in the evaluation of machine vision tasks, for situations where false detection is more concerned, precision can be selected as the evaluation index, for situations where missed detection is more concerned, recall can be selected as the evaluation index, and for situations where both false detection and missed detection are more concerned, F value can be selected as the evaluation index.
[0041] The four indicators of accuracy, precision, recall and F-value require the following four benchmark statistical values: TP (True Positive) is the number of positive samples judged as positive samples; FN (False Negative) is the number of positive samples misjudged as negative samples; FP (False Positive) is the number of negative samples misjudged as positive samples; TN (True Negative) is the number of negative samples judged as negative samples. Precision rate indicates the correct probability of judging as positive samples, so the precision rate is:
[0042]
[0043] Accuracy refers to the probability of making a correct judgment among all judgments:
[0044]
[0045] The recall rate indicates the probability of being judged correctly in the positive sample. There are two possibilities for judging the positive sample, one is to judge the positive sample as a positive sample (TP), and the other is to judge the positive sample as a negative sample (FN). Therefore, the recall rate is expressed as:
[0046]
[0047] The F value is the harmonic mean of precision and recall:
[0048]
[0049] In step S106, when the evaluation index value of the neural network reaches a preset value, the data volume of the compressed feature value corresponding to the candidate layer is determined as a reference index for the candidate layer.
[0050] In some embodiments, the data amount of the compressed characteristic value of each image corresponding to the candidate layer is summed or averaged as the data amount of the compressed characteristic value corresponding to the candidate layer.
[0051] In step S108, a candidate layer is selected as a layer to be compressed according to reference indicators of each candidate layer, so that the characteristic value output by the layer to be compressed is compressed in the process of using the neural network.
[0052] In some embodiments, based on the reference indicators of each candidate layer, an alternative layer with the smallest reference indicator is selected as the layer to be compressed. In some embodiments, the layer to be compressed in the neural network and the layers connected before the layer to be compressed are deployed at the encoding end, and the layer connected after the layer to be compressed is deployed at the decoding end. For example, the encoding end is located at the edge computing node (such as a sensor), and the decoding end is located in the cloud (such as an intelligent platform). For deep learning feature coding for machine vision, the encoding end extracts the feature coding of the video image, compresses it, and sends it to the decoding end for decoding and subsequent processing. In the manner in which the encoding end and the decoding end are deployed separately, the feature coding can be calculated after the edge node collects the video, and then transmitted to the cloud for subsequent processing. According to the method of the above embodiment, an alternative layer with the smallest reference indicator is selected as the layer to be compressed, which minimizes the amount of data transmitted between the encoding end and the decoding end, and reduces the transmission delay and occupied bandwidth resources.
[0053] In the above embodiment, for each candidate layer in the neural network, the characteristic value of the image output by the candidate layer is compressed in a preset manner to obtain a compressed characteristic value. After the compressed characteristic value is processed in the subsequent layers of the neural network, an evaluation index value for evaluating the performance of the neural network can be obtained. If the evaluation index value of the neural network reaches the preset value, the data volume of the compressed characteristic value corresponding to the candidate layer is determined as the reference index of the candidate layer, and an candidate layer is selected as the layer to be compressed according to the reference index of each candidate layer. In the subsequent use of the neural network, the characteristic value of the image output by the layer to be compressed is compressed and then input into the subsequent layers of the neural network for processing. The method of the above embodiment can compress the data of the image processed by the neural network when the evaluation index of the neural network meets the task requirements, thereby improving the computational efficiency of the neural network. When the method of the above embodiment is applied to deep learning feature coding for machine vision, the compression rate and encoding efficiency of video data can be improved under the premise of meeting the machine vision task, and the computational load can be saved.
[0054] Combine the following Figure 2 Other embodiments of the image processing method disclosed herein are described.
[0055] Figure 2 Flowcharts of other embodiments of the image processing method disclosed in the present invention. Figure 2 As shown, the method of this embodiment includes: steps S202 to S224.
[0056] In step S202, a list of candidate layers and multiple images in the neural network are obtained, and the current bit depth value is initialized to a preset minimum value.
[0057] In step S204, the first candidate layer is selected as the current candidate layer.
[0058] In step S206, the image is input into the neural network.
[0059] In step S208, the characteristic values of the image output by the current candidate layer are compressed according to the current bit depth value to obtain compressed characteristic values.
[0060] In step S210, the compressed feature value is input into the layer connected after the candidate layer to determine the evaluation index value of the neural network.
[0061] In step S212, it is determined whether the evaluation index value of the neural network reaches a preset value. If not, step S214 is executed; otherwise, step S218 is executed.
[0062] In step S214, it is determined whether the current bit depth value reaches a preset maximum value, if not, step S216 is executed, otherwise step S220 is executed.
[0063] In step S216, the current bit depth value is increased by a preset step size to be used as the current bit depth value, and the process returns to step S208 and starts again.
[0064] For example, add 1 to the current BD value.
[0065] In step S218, the data volume of the compressed characteristic value corresponding to the current candidate layer is determined as a reference indicator for the current candidate layer.
[0066] In step S220, it is determined whether the current candidate layer is the last candidate layer in the candidate layer list, if not, step S222 is executed, otherwise, step S224 is executed.
[0067] In step S222, the next candidate layer is selected as the current feature layer, the current bit depth value is assigned to the preset minimum value, and the process returns to step S206 and starts again.
[0068] In step S224, a candidate layer is selected as the layer to be compressed according to the reference indicators of each candidate layer.
[0069] The following describes a specific application example using the neural network ResNe101 as an example. ResNe101 is an FPN network, and the feature layer entering the horizontal connection is the conv2_x / conv3_x / conv4_x / conv5_x output layer. Therefore, the candidate layer can be the layer before the conv2_x output layer.
[0070] 1. Randomly select M images, determine the candidate layer list as (conv2_x_maxpool, conv2_x_block1, conv2_x_block2, conv2_x_block3), extract the feature values of the candidate layers, select the recall rate as the evaluation index A of the network, and the preset value Thres corresponding to A is set to 90%.
[0071] 2. The sequence number i of the candidate layer is initialized to 0, and M images are input into ResNe101.
[0072] 3. The eigenvalues output by the i-th candidate layer are compressed according to different BD values, where the BD value range is an integer from 3 to 10. The evaluation index value A obtained by the subsequent layers is input into the compressed eigenvalues.
[0073] 4. When the evaluation index A reaches the preset value Thres, that is, A l=i,BD=d When the value is ≥90%, the calculation is stopped, and the data volume of the compressed characteristic value under the current BD of the current candidate layer is calculated as a reference indicator, for example, the calculation is performed using the following formula.
[0074]
[0075] 5. When the evaluation index value corresponding to the i-th candidate layer reaches the preset value, record the reference index Q of the candidate layer l=i,BD=d , the BD value is d; otherwise, the layer information is not recorded.
[0076] 6. Increase the sequence number i by 1. If the list length has reached 4, go to step 7; if the list length has not reached 4, go to step 3.
[0077] 7. Compare the reference indicators of each candidate layer, extract the layer with the smallest reference indicator as the feature extraction layer, and obtain the corresponding BD value.
[0078] Combine the following Figure 3 Some further embodiments of the image processing method disclosed herein are described.
[0079] Figure 3 Flowcharts of some other embodiments of the image processing method disclosed herein. Figure 3 As shown, the method of this embodiment includes: steps S302 to S306.
[0080] In step S302, the image to be processed is input into the neural network.
[0081] The image to be processed is, for example, an image of a video in deep learning feature coding for machine vision.
[0082] In step S304, after the image to be processed passes through the layer to be compressed, the characteristic value of the image to be processed output by the layer to be compressed is compressed according to a preset method to obtain a compressed characteristic value.
[0083] For example, the above embodiment describes that the BD value corresponding to the layer to be compressed will be recorded, and the characteristic value of the image to be processed outputted from the layer to be compressed is compressed according to the recorded BD value. The specific compression method can refer to the above embodiment.
[0084] In step S306, the compressed feature values are input into the layer connected after the layer to be compressed to obtain the output result of the neural network.
[0085] In some embodiments, the layer to be compressed in the neural network and the layers connected before the layer to be compressed are deployed at the encoding end, and the layers connected after the layer to be compressed are deployed at the decoding end. For example, the encoding end is located at the edge computing node, and the decoding end is located in the cloud. The encoding end encodes the compressed feature values (for example, converts them into binary using entropy coding) and sends them to the decoding end; after decoding, the decoding end inputs the compressed feature values into the layer connected after the layer to be compressed. That is, after the encoding end collects the video image and completes the deep learning feature coding for machine vision, it compresses it and sends it to the decoding end for subsequent video processing.
[0086] The present disclosure also provides an image processing device, Figure 4 Give a description.
[0087] Figure 4 FIG. 1 is a structural diagram of some embodiments of the image processing device disclosed in the present invention. Figure 4 As shown, the device 40 of this embodiment includes: a compression module 410 , an evaluation index determination module 420 , a reference index determination module 430 , and a selection module 440 .
[0088] The compression module 410 is used to compress the characteristic value of the image output by each candidate layer in the multiple candidate layers in the neural network according to a preset method to obtain a compressed characteristic value.
[0089] In some embodiments, when the neural network is a feature pyramid network FPN, the candidate layers are selected from the layers before the layer that first enters the FPN lateral connection.
[0090] The evaluation index determination module 420 is used to input the compressed feature value into the layer connected after the candidate layer to determine the evaluation index value of the neural network.
[0091] In some embodiments, the evaluation index value of the neural network includes: accuracy, precision, recall rate or F value.
[0092] In some embodiments, the compression module 410 is used to compress the characteristic value of the image output by the alternative layer according to the current bit depth value to obtain a compressed characteristic value, wherein the current bit depth value is greater than or equal to a preset minimum value. The evaluation index determination module 420 is also used to increase the current bit depth value by a preset step size when the evaluation index value of the neural network does not reach a preset value, and return the value compression module 410 as the current bit depth value. The compression module 410 is used to re-compress the characteristic value of the image output by the alternative layer according to the current bit depth value and start executing until the current bit depth value reaches a preset maximum value, or the evaluation index value of the neural network reaches a preset value.
[0093] In some embodiments, the compression module 410 is used to determine the current scale based on the characteristic value of the image output by the alternative layer and the current bit depth value; quantize the characteristic value of the image output by the alternative layer according to the current scale; multiply the quantized value by the scale to obtain a compressed characteristic value.
[0094] The reference index determination module 430 is used to determine the data volume of the compressed feature value corresponding to the candidate layer as the reference index of the candidate layer when the evaluation index value of the neural network reaches a preset value.
[0095] In some embodiments, the data amount of the compressed characteristic value of each image corresponding to the candidate layer is summed or averaged as the data amount of the compressed characteristic value corresponding to the candidate layer.
[0096] The selection module 440 is used to select a candidate layer as a layer to be compressed according to the reference indicators of each candidate layer, so as to compress the characteristic value output by the layer to be compressed in the process of using the neural network.
[0097] In some embodiments, the selection module 440 is used to select a candidate layer with the smallest reference index as the layer to be compressed according to the reference indexes of the candidate layers.
[0098] In some embodiments, the image processing device 40 also includes: an image processing module 450, which is used to input the image to be processed into the neural network; after the image to be processed passes through the layer to be compressed, the feature value of the image to be processed output by the layer to be compressed is compressed according to a preset method to obtain a compressed feature value; the compressed feature value is input into the layer connected after the layer to be compressed to obtain the output result of the neural network.
[0099] In some embodiments, the layer to be compressed and the layers connected before the layer to be compressed in the neural network are deployed at the encoding end, and the layers connected after the layer to be compressed are deployed at the decoding end; the encoding end includes an image processing device 40. The image processing module 450 is used to encode the compressed feature value and send it to the decoding end, so that the decoding end inputs the compressed feature value into the layer connected after the layer to be compressed after decoding.
[0100] In some embodiments, the encoding end is located at an edge computing node and the decoding end is located at the cloud.
[0101] The image processing apparatus in the embodiments of the present disclosure may be implemented by various computing devices or computer systems. Figure 5 as well as Figure 6 Give a description.
[0102] Figure 5 FIG. 1 is a structural diagram of some embodiments of the image processing device disclosed in the present invention. Figure 5 As shown, the device 50 of this embodiment includes: a memory 510 and a processor 520 coupled to the memory 510 , and the processor 520 is configured to execute the image processing method in any of some embodiments of the present disclosure based on the instructions stored in the memory 510 .
[0103] The memory 510 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.
[0104] Figure 6 FIG. 1 is a structural diagram of some other embodiments of the image processing device disclosed in the present invention. Figure 6 As shown, the device 60 of this embodiment includes: a memory 610 and a processor 620, which are similar to the memory 510 and the processor 520 respectively. It can also include an input and output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650 and the memory 610 and the processor 620 can be connected, for example, through a bus 660. Among them, the input and output interface 630 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 640 provides a connection interface for various networked devices, for example, it can be connected to a database server or a cloud storage server. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives.
[0105] The present disclosure also provides an image processing system, Figure 7 Give a description.
[0106] Figure 7 FIG. 1 is a structural diagram of some embodiments of the image processing system disclosed in the present invention. Figure 7 As shown, the system 7 of this embodiment includes: an encoding end 72 and a decoding end 74. The encoding end 72 includes the image processing device 40 / 50 / 60. The layers to be compressed and the layers connected before the layers to be compressed in the neural network are deployed at the encoding end 72, and the layers connected after the layers to be compressed are deployed at the decoding end 74.
[0107] In some embodiments, the encoding end 72 is located at an edge computing node, and the decoding end 74 is located at the cloud. The evaluation index determination module 420 is used to send the compressed feature value to the decoding end 74, and the decoding end 74 inputs the compressed feature value into the layer connected after the candidate layer, determines the evaluation index value of the neural network and returns it to the evaluation index determination module 420.
[0108] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0109] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0112] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.
Claims
1. An image processing method, comprising: For each candidate layer in the plurality of candidate layers in the neural network, compressing the characteristic value of the image output by the candidate layer in a preset manner to obtain a compressed characteristic value, including: compressing the characteristic value of the image output by the candidate layer according to a current bit depth value to obtain a compressed characteristic value, wherein the current bit depth value is greater than or equal to a preset minimum value; Inputting the compressed feature value into a layer connected after the candidate layer to determine an evaluation index value of the neural network; When the evaluation index value of the neural network reaches a preset value, determining the data amount of the compressed feature value corresponding to the candidate layer as a reference index of the candidate layer; Selecting a candidate layer as a to-be-compressed layer according to reference indicators of each candidate layer, so as to compress the characteristic value output by the to-be-compressed layer in the process of using the neural network; When the evaluation index value of the neural network does not reach a preset value, the current bit depth value is increased by a preset step length as the current bit depth value; The process starts again by compressing the characteristic value of the image output by the candidate layer according to the current bit depth value, until the current bit depth value reaches a preset maximum value, or the evaluation index value of the neural network reaches a preset value.
2. The image processing method according to claim 1, wherein: The step of determining the amount of data of the compressed characteristic value corresponding to the candidate layer comprises: The data amount of the compressed characteristic value of each image corresponding to the candidate layer is summed or averaged as the data amount of the compressed characteristic value corresponding to the candidate layer.
3. The image processing method according to claim 1, wherein: The selecting a candidate layer as the layer to be compressed according to the reference indicators of each candidate layer comprises: According to the reference indicators of each candidate layer, a candidate layer with the smallest reference indicator is selected as the layer to be compressed.
4. The image processing method according to claim 1, wherein: In the case where the neural network is a feature pyramid network FPN, the candidate layer is selected from the layers before the layer that first enters the FPN lateral connection.
5. The image processing method according to claim 1, further comprising: Inputting the image to be processed into the neural network; After the image to be processed passes through the layer to be compressed, the characteristic value of the image to be processed output by the layer to be compressed is compressed according to a preset method to obtain a compressed characteristic value; The compressed feature value is input into the layer connected after the layer to be compressed to obtain the output result of the neural network.
6. The image processing method according to claim 5, wherein: The layer to be compressed and the layers connected before the layer to be compressed in the neural network are deployed at the encoding end, and the layers connected after the layer to be compressed are deployed at the decoding end; The layer connected after inputting the compressed feature value into the layer to be compressed comprises: The encoding end encodes the compressed feature value and sends it to the decoding end; After decoding, the decoding end inputs the compressed feature value into the layer connected after the layer to be compressed.
7. The image processing method according to claim 6, wherein: The encoding end is located at the edge computing node, and the decoding end is located at the cloud.
8. The image processing method according to claim 1, wherein: The evaluation index values of the neural network include: accuracy, precision, recall or F value.
9. The image processing method according to claim 1, wherein: The step of compressing the characteristic value of the image output by the candidate layer according to the current bit depth value to obtain the compressed characteristic value comprises: Determine the current magnitude according to the characteristic value of the image output by the candidate layer and the current bit depth value; Quantize the feature value of the image output by the candidate layer according to the current magnitude; The quantized value is multiplied by the magnitude to obtain a compressed eigenvalue.
10. An image processing device, comprising: A compression module, for compressing, for each candidate layer in the plurality of candidate layers in the neural network, a feature value of an image output by the candidate layer in a preset manner to obtain a compressed feature value, comprising: compressing the feature value of the image output by the candidate layer according to a current bit depth value to obtain a compressed feature value, wherein the current bit depth value is greater than or equal to a preset minimum value; An evaluation index determination module, used for inputting the compressed feature value into a layer connected after the candidate layer to determine the evaluation index value of the neural network; A reference index determination module, used to determine the amount of data of the compressed feature value corresponding to the candidate layer as a reference index of the candidate layer when the evaluation index value of the neural network reaches a preset value; A selection module, used for selecting a candidate layer as a to-be-compressed layer according to reference indicators of each candidate layer, so as to compress the characteristic value output by the to-be-compressed layer in the process of using the neural network; Among them, the evaluation index determination module is also used to increase the current bit depth value by a preset step size when the evaluation index value of the neural network does not reach a preset value, and return a value compression module as the current bit depth value. The compression module is used to re-compress the characteristic value of the image output by the alternative layer according to the current bit depth value and start execution until the current bit depth value reaches a preset maximum value, or the evaluation index value of the neural network reaches a preset value.
11. The image processing apparatus according to claim 10, further comprising: An image processing module, used for inputting an image to be processed into the neural network; After the image to be processed passes through the layer to be compressed, the characteristic value of the image to be processed output by the layer to be compressed is compressed according to a preset method to obtain a compressed characteristic value; The compressed feature value is input into the layer connected after the layer to be compressed, so as to obtain the output result of the neural network.
12. An image processing device, comprising: processor; as well as A memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the image processing method according to any one of claims 1 to 9.
13. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
14. An image processing system, comprising: An encoding end and a decoding end, wherein the encoding end comprises the image processing device according to any one of claims 10 to 12, the layer to be compressed and the layers connected before the layer to be compressed in the neural network are deployed at the encoding end, and the layer connected after the layer to be compressed is deployed at the decoding end.
Citation Information
Patent Citations
Image encoding method and corresponding terminal
CN110139109A
Image compression method, device and system
CN111050174A
Searching method of machine learning model and related device, and equipment
CN111178546A
Video compression processing method and device, storage medium and electronic equipment
CN111263161A
Image processing method and device, terminal equipment and computer readable storage medium
CN111383289A
