Image processing method and device, computer device and storage medium
By compiling neural network models into directed graphs and optimizing them, the problem of limited operating scenarios for neural network models is solved, and efficient image processing under different data shapes is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DAWNING INFORMATION IND (BEIJING) CO LTD
- Filing Date
- 2022-11-10
- Publication Date
- 2026-06-26
Smart Images
Figure CN115688596B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine learning technology, and in particular to an image processing method, apparatus, computer device, and storage medium. Background Technology
[0002] With the development of neural network technology, the inference framework technology used to perform neural network inference is also constantly evolving. MIGraphX is an inference framework used to accelerate machine learning model inference. Typically, before executing the neural network model, the MIGraphX framework first compiles the neural network model into a complete graph, and then uses the compiled graph to process the data. The size of the input data for this neural network model is usually fixed.
[0003] However, since the size of the input data for this neural network model is usually fixed, the scenarios in which the neural network model can be run are limited. Summary of the Invention
[0004] Therefore, it is necessary to provide an image processing method, apparatus, computer device, and storage medium that can expand the operating scenarios of neural network models and improve the operating performance of neural network models when the size of input data is not fixed, in order to address the above-mentioned technical problems.
[0005] Firstly, this application provides an image processing method. The method includes:
[0006] Based on the maximum data shape corresponding to the image set to be processed, the preset neural network model is compiled into a directed graph;
[0007] If the target data shape is no larger than the maximum data shape, the directed graph is optimized based on the target data shape to obtain the optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model.
[0008] The optimized directed graph is used to process the target image to obtain the corresponding processing result.
[0009] In one embodiment, the directed graph is optimized based on the target data shape to obtain an optimized directed graph, including:
[0010] Invoke the preset optimization method;
[0011] By using optimization methods and the target data shape, the directed graph is optimized to obtain the optimized directed graph.
[0012] In one embodiment, the directed graph is optimized using an optimization method and a target data shape to obtain an optimized directed graph, including:
[0013] By using optimization methods and target data shape, the operators and compile time corresponding to each node of the directed graph are optimized to obtain the optimized directed graph.
[0014] In one embodiment, the method further includes:
[0015] The maximum data shape is determined based on the attribute information of each image in the image set to be processed; the attribute information includes the image size and the image resolution.
[0016] In one embodiment, the method further includes:
[0017] The maximum data shape is determined by using the constraints corresponding to the neural network model.
[0018] In one embodiment, a preset neural network model is compiled into a corresponding directed graph based on the maximum data shape corresponding to the image set to be processed, including:
[0019] Based on the maximum data shape, the layers of the neural network model are used as nodes, and the connections between the layers are used as edges to construct a directed graph.
[0020] Secondly, this application also provides an image processing apparatus. The apparatus includes:
[0021] The compilation module is used to compile a preset neural network model into a corresponding directed graph based on the maximum data shape corresponding to the image set to be processed.
[0022] The optimization module is used to optimize the directed graph based on the target data shape when the target data shape is less than or equal to the maximum data shape, and obtain the optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model;
[0023] The processing module is used to process the target image using the optimized directed graph to obtain the processing result corresponding to the target image.
[0024] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0025] Based on the maximum data shape corresponding to the image set to be processed, the preset neural network model is compiled into a directed graph;
[0026] If the target data shape is no larger than the maximum data shape, the directed graph is optimized based on the target data shape to obtain the optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model.
[0027] The optimized directed graph is used to process the target image to obtain the corresponding processing result.
[0028] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0029] Based on the maximum data shape corresponding to the image set to be processed, the preset neural network model is compiled into a directed graph;
[0030] If the target data shape is no larger than the maximum data shape, the directed graph is optimized based on the target data shape to obtain the optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model.
[0031] The optimized directed graph is used to process the target image to obtain the corresponding processing result.
[0032] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:
[0033] Based on the maximum data shape corresponding to the image set to be processed, the preset neural network model is compiled into a directed graph;
[0034] If the target data shape is no larger than the maximum data shape, the directed graph is optimized based on the target data shape to obtain the optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model.
[0035] The optimized directed graph is used to process the target image to obtain the corresponding processing result.
[0036] The aforementioned image processing method, apparatus, computer equipment, and storage medium, based on the maximum data shape corresponding to the set of images to be processed, can compile and optimize a preset neural network model into a directed graph. By using the data shape corresponding to the target image to be processed in the current stage of the neural network model as the target data shape, the directed graph can be optimized based on the target data shape, provided the target data shape is not larger than the maximum data shape. Since the optimized directed graph is optimized based on the data shape corresponding to the target image to be processed in the current stage, and the data shape of the target image to be processed in the current stage is definite, this allows for explicit optimization of the directed graph. The resulting optimized directed graph can adapt to the shape of the target image to be processed in the current stage, enabling it to process the target image and obtain the corresponding processing result. In other words, by optimizing the directed graph based on the data shape corresponding to the target image to be processed in the current stage, the optimized directed graph can adapt to target images with different data shapes, enabling the processing of target images with different data shapes. This expands the operating scenarios of the neural network model and improves its performance when the shape of the input target image is not fixed. Attached Figure Description
[0037] Figure 1 This is a flowchart illustrating an image processing method in one embodiment;
[0038] Figure 2 This is a flowchart illustrating the image processing method in another embodiment;
[0039] Figure 3 This is a flowchart illustrating the image processing method in another embodiment;
[0040] Figure 4 This is a flowchart illustrating the image processing method in another embodiment;
[0041] Figure 5 This is a structural block diagram of an image processing device in one embodiment;
[0042] Figure 6 This is a structural block diagram of an image processing device in one embodiment;
[0043] Figure 7 This is a structural block diagram of an image processing device in one embodiment;
[0044] Figure 8 This is a structural block diagram of an image processing device in one embodiment;
[0045] Figure 9 This is a structural block diagram of an image processing device in one embodiment. Detailed Implementation
[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0047] It should be noted that MIGraphX is an inference framework used to accelerate machine learning model inference. Typically, to improve inference performance, MIGraphX compiles the neural network model into a single graph before executing it, and then uses this compiled graph to process the data. However, currently, MIGraphX only supports the compilation of neural network models with fixed-size input data (i.e., static shape mode). MIGraphX employs many optimization methods during the compilation phase to improve performance, some of which require knowledge of the explicit shape information of the input data, such as constant propagation and constant folding optimization, memory reuse optimization, and operator fusion optimization. Memory reuse optimization refers to the ability to optimize memory reuse in static shape models, where the shape information of the data processed by the model is explicit. This is achieved by analyzing the data dependencies in the computation graph. For example, consider a 6-layer deep learning model, denoted as ABCDEF, with data dependencies A->B->C->D->E->F. Assuming each layer requires 100MB of memory, without optimization, the model would require 600MB. However, with optimization, analyzing the dependencies reveals that B depends on A's data, so B cannot reuse A's memory. Therefore, 100MB needs to be allocated to A, and 100MB to B. Since C has no dependency on A, it can reuse A's 100MB, and so on. DEF can reuse their previous memory, reducing the total memory requirement to only 200MB. This significantly reduces memory consumption. Since most deep learning models have far more than 6 layers, the effectiveness of memory reuse optimization becomes more pronounced with increasing layer count. For operator optimization, taking the resize operator as an example, when the resize operator uses the nearest neighbor interpolation algorithm, the MIGraphX framework optimizes its implementation. MIGraphX uses the gather operator to implement the resize operator, thereby improving performance. In static shape mode, since the input and output sizes are known, if nearest neighbor interpolation is used, the index of the input image corresponding to each pixel in the output image is determined. Therefore, the index value can be calculated in advance at compile time and saved as a constant as the input to the gather operator. At runtime, the gather operator directly retrieves data according to the index value, thus improving the performance of the resize operator. However, the above optimization methods require explicit shape information to be known at compile time, while dynamic shape mode cannot know explicit shape information. Therefore, the above optimizations cannot be performed in dynamic shape mode, thus limiting the operating scenarios of neural network models.
[0048] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown. Figure 1 A computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 1 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores image data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an image processing method.
[0049] In one embodiment, such as Figure 2 As shown, an image processing method is provided, which is applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:
[0050] S201, compile the preset neural network model into a directed graph according to the maximum data shape corresponding to the image set to be processed.
[0051] The image set to be processed can be a set of medical images or a set of various images collected in a real-world scenario, such as a set of real-time acquired road images. Optionally, in this embodiment, the image set to be processed can be acquired in real time from an image acquisition device or from a preset memory. Optionally, the neural network model in this embodiment can be a neural network model such as a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN).
[0052] Optionally, in this embodiment, since the maximum data shape corresponding to the acquired image set to be processed is a fixed shape, it can be assumed that the preset neural network model is compiled in static graph mode, and the preset neural network model is compiled into a directed graph. Here, a directed graph refers to a graph containing nodes and edges. It can be understood that the directed graph in this embodiment is obtained by compiling the neural network model. Therefore, the node and edge information of the directed graph in this embodiment can be used to characterize the network structure information of the neural network model.
[0053] Optionally, in this embodiment, the maximum data shape can be determined based on the parameters of each image in the image set to be processed by the neural network model, such as the pixel information of the image. Alternatively, a certain range can be defined for the maximum data shape based on the parameters of each image in the image set to be processed by the neural network model, and the maximum data shape can be determined from that range.
[0054] Furthermore, based on traditional techniques, when compiling a neural network model in static graph mode, various optimization methods are employed during the compilation phase using explicit shape information to improve the performance of the neural network model. As an optional implementation method, in this embodiment, when compiling a preset neural network model based on the maximum data shape corresponding to the image set to be processed, the aforementioned memory reuse optimization and operator fusion optimization methods can also be used to improve the performance of the neural network model based on the aforementioned maximum data shape.
[0055] S202, if the target data shape is not greater than the maximum data shape, optimize the directed graph based on the target data shape to obtain the optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model.
[0056] In this embodiment, the image set to be processed in the preset neural network model contains images with various data shapes. The directed graphs corresponding to images with different data shapes are different. Therefore, in this embodiment, when processing the target image to be processed in the current stage, the target data shape corresponding to the target image to be processed in the current stage can be compared with the aforementioned maximum data shape. If the target data shape of the target image to be processed in the current stage is not greater than the maximum data shape, the computer device optimizes the aforementioned directed graph based on the target data shape of the target image to be processed in the current stage, obtaining the directed graph corresponding to the target image to be processed in the current stage. For example, the aforementioned target data shape can be determined based on the size and resolution of the target image to be processed in the current stage. Optionally, in this embodiment, the optimization of the directed graph based on the target data shape can include operator optimization and compile-time optimization, etc. Furthermore, it should be noted that in this embodiment, if the target data shape corresponding to the target image to be processed in the current stage is greater than the aforementioned maximum data shape, the computer device will report an error and exit execution.
[0057] S203, the target image is processed using the optimized directed graph to obtain the processing result corresponding to the target image.
[0058] Optionally, in this embodiment, if the above-mentioned neural network model is a segmentation model, the computer device can use the optimized directed graph to segment the target image and obtain the segmentation result corresponding to the target image; or, if the above-mentioned neural network model is a classification model, the computer device can use the optimized directed graph to classify the target image and obtain the classification result corresponding to the target image.
[0059] Additionally, it's understandable that since the maximum data shape corresponding to the image set to be processed provides a clear data shape, the amount of memory required for each node in the neural network model can be known during compilation. This allows memory to be allocated for each node in advance. Furthermore, since the memory allocated to each node is the maximum during program execution, there is no need to allocate memory again, avoiding frequent memory allocation and deallocation, thus improving system performance. Moreover, because the maximum data shape corresponding to the image set to be processed provides a clear data shape, memory reuse optimization can be performed by analyzing the data dependencies between layers of the neural network model, which can significantly reduce memory consumption.
[0060] In the image processing method described above, based on the maximum data shape corresponding to the set of images to be processed, a preset neural network model can be compiled and optimized into a directed graph. The data shape corresponding to the target image to be processed in the current stage of the neural network model is taken as the target data shape. Thus, the directed graph can be optimized based on the target data shape when the target data shape is not greater than the maximum data shape, resulting in an optimized directed graph. Since the optimized directed graph is optimized based on the data shape corresponding to the target image to be processed in the current stage, and the data shape corresponding to the target image to be processed in the current stage is clear, the directed graph can be optimized explicitly. The resulting optimized directed graph can adapt to the shape of the target image to be processed in the current stage, enabling the optimized directed graph to process the target image and obtain the corresponding processing result. In other words, by optimizing the directed graph based on the data shape corresponding to the target image to be processed in the current stage, the optimized directed graph can adapt to target images with different data shapes, enabling the processing of target images with different data shapes. This expands the operating scenarios of the neural network model and improves the operating performance of the neural network model when the shape of the input target image is not fixed.
[0061] In the scenario where the target data shape is no larger than the maximum data shape, and the directed graph is optimized based on the target data shape, this embodiment will provide a detailed explanation of the optimization process for the directed graph. In one embodiment, such as Figure 3 As shown, the above S202 includes:
[0062] S301, invoke the preset optimization method.
[0063] In this embodiment, when the shape of the input image to the neural network model is not fixed, the operation of the neural network model will change, requiring dynamic modification of the directed graph to adapt to different input shapes. Optionally, the optimization method in this embodiment can be obtained by pre-encapsulating all operations that need to be performed when the shape of the model's input data is not fixed. For example, the operations that need to be performed when the shape of the model's input data is not fixed may include modifying operator attributes, modifying compile-time optimizations, etc. It is understood that by pre-encapsulating all operations that need to be performed when the shape of the model's input data is not fixed into the above optimization method, when the shape of the neural network model's input data is fixed, this method does not need to be called, and it will not affect the operation of the neural network model when the shape of the input data is fixed; when the shape of the neural network model's input data is not fixed, and optimization of the compiled directed graph is required, only this optimization method needs to be called. Optionally, in this embodiment, the computer device can send a call instruction to the memory storing the optimization method, and call the optimization method from the memory based on the call instruction.
[0064] S302, using optimization methods and target data shape, optimize the directed graph to obtain the optimized directed graph.
[0065] Optionally, in this embodiment, a preset optimization method and target data shape can be used to optimize the operators and compilation time corresponding to each node of the directed graph to obtain an optimized directed graph.
[0066] Optionally, taking the broadcast operator as an example, assuming the input data shape of the neural network model is [8,3,224,224], this data shape represents 8 images of size 224x224 with 3 channels. Assume the first layer of this neural network model is a convolutional layer with a bias term, and the kernel size of this convolutional layer is [32,3,3,3], representing 32 convolutional kernels of size 3x3 with 3 channels. The space between the element bounding box and the element content of this convolutional kernel is 1. The size of the result obtained after performing convolution through this kernel is [8,32,224,224]. Then, for each convolutional kernel in the convolutional layer of the convolutional neural network, a bias term is assigned (which can be...). (Understanding it as a constant), the bias of convolution in MIGraphX is achieved by adding a broadcast bias term to the convolution result. The specific formula is: result = conv(input) + broadcast(bias). First, the bias term is broadcast to the convolution result of size [8, 32, 224, 224], and then added to the convolution result to obtain the final result. The broadcast operator in MIGraphX has an attribute, which is the size of the bias term after broadcasting. In this example, the value of this attribute is [8, 32, 224, 224]. In modes where the shape of the input data is not fixed, if the shape of the input changes, this attribute of the broadcast needs to be modified.
[0067] Optionally, MIGraphX converts global pooling operators into reshape and reduce operators at compile time, which improves the performance of these operators. For example, in the ResNet50 network, there is a final global pooling operation. Suppose the input size of this global pooling is [1,2048,7,7], and the result size after this global pooling is [1,2048,1,1]. After conversion to reshape and reduce operators, the reshape operator first needs to convert the global pooling input size [1,2048,7,7] to [2048,49], and then after the reduce operator, it becomes [1,2048,1,1]. The reshape operator in MIGraphX has an attribute representing the reshape size. In this example, the value of this attribute is [2048,49]. In modes where the input data shape is not fixed, if the input shape changes, the attribute value of the reshape operator in this compile-time optimization needs to be modified.
[0068] In this embodiment, by calling a preset optimization method and utilizing the optimization method and the target data shape corresponding to the target image to be processed in the current stage of the neural network model, the directed graph is optimized. This enables the optimized directed graph to adapt to the data shape of the target image to be processed in the current stage, allowing the neural network model to process different images with variable input data shapes. This expands the application scenarios of the neural network model and improves its performance.
[0069] In the scenario described above, where a pre-defined neural network model is compiled into a directed graph based on the maximum data shape corresponding to the image set to be processed, it is necessary to first determine the maximum data shape corresponding to the image set to be processed. The following will explain in detail two methods for determining the maximum data shape corresponding to the image set to be processed:
[0070] Method 1: In some scenarios, if the attribute information of each image in the image set to be processed is known, the attribute information of each image in the image set to be processed can be determined. In one embodiment, the above method further includes: determining the maximum data shape based on the attribute information of each image in the image set to be processed; the attribute information includes the size of the image and the resolution of the image.
[0071] In this embodiment, the computer device can determine the maximum data shape based on the size and resolution of each image in the image set to be processed by the neural network. For example, the computer device can determine the maximum data shape as the average size and average resolution of each image in the image set to be processed, or the computer device can determine the maximum data shape as the maximum size and maximum resolution of each image in the image set to be processed.
[0072] In this embodiment, since the attribute information of each image in the image set to be processed is clear, the maximum data shape corresponding to the image set to be processed can be accurately determined based on the attribute information of each image in the image set to be processed, thereby improving the accuracy of the determined maximum data shape and thus improving the accuracy of the directed graph compiled by the preset neural network model based on the maximum data shape.
[0073] Method 2: In some scenarios, if the attribute information of each image in the image set to be processed is uncertain, in one embodiment, the above method further includes: using the constraints corresponding to the neural network model to determine the maximum data shape.
[0074] Optionally, the constraints corresponding to the neural network model may include the number of layers in the neural network model, the maximum size of the image that the neural network model can process, the maximum resolution of the image that the neural network model can process, etc. Taking the number of layers in the neural network model as an example, the computer device can determine the above-mentioned maximum data shape based on the number of layers in the neural network model.
[0075] It is understandable that, in this embodiment, after determining the maximum data shape corresponding to the neural network model, memory can be pre-allocated for each node of the directed graph corresponding to the neural network model, eliminating the need to allocate memory during program execution, thus avoiding frequent memory allocation and deallocation and improving system performance.
[0076] In this embodiment, even when the attribute information of each image in the image set to be processed is uncertain, the maximum data shape corresponding to the image set to be processed by the neural network model can be determined by the constraints corresponding to the neural network model, thereby improving the efficiency of determining the maximum data shape corresponding to the image set to be processed by the neural network model.
[0077] In the scenario described above, where a preset neural network model is compiled into a corresponding directed graph based on the maximum data shape corresponding to the image set to be processed, in one embodiment, S201 includes: constructing a directed graph by taking each layer of the neural network model as nodes and the connection relationship between each layer as edges, based on the maximum data shape.
[0078] It should be noted that a neural network model is a complex network system formed by extensive interconnection of a large number of simple processing units. A neural network model may include multiple layers, such as input layers, convolutional layers, and output layers. In this embodiment, the computer device can construct a directed graph corresponding to the neural network model by taking each layer structure in the neural network model as nodes and the connection relationship between each layer structure as edges, according to the maximum data shape.
[0079] In this embodiment, based on the maximum data shape corresponding to the image set to be processed, the preset neural network model can be accurately compiled and optimized into the corresponding directed graph, which is beneficial for the neural network model to further process the image set and improves the model's running performance.
[0080] The following describes an embodiment of this disclosure using a specific image processing scenario, such as... Figure 4 As shown, the method includes the following steps:
[0081] S1. Determine the maximum data shape based on the size and resolution of each image in the image set to be processed; or determine the maximum data shape using the constraints corresponding to the neural network model.
[0082] S2. Based on the maximum data shape, construct a directed graph by taking each layer of the neural network model as nodes and the connections between each layer as edges.
[0083] S3, the shape of the target data corresponding to the target image to be processed in the current stage of the neural network model.
[0084] S4. Call the preset optimization method; use the optimization method and the target data shape to optimize the operators and compiler corresponding to each node of the directed graph to obtain the optimized directed graph.
[0085] S5. Use the optimized directed graph to process the target image and obtain the processing result corresponding to the target image.
[0086] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0087] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.
[0088] In one embodiment, such as Figure 5 As shown, an image processing apparatus is provided, including: a compilation module 10, an optimization module 11, and a processing module 12, wherein:
[0089] The compilation module 10 is used to compile a preset neural network model into a corresponding directed graph based on the maximum data shape corresponding to the image set to be processed.
[0090] Optimization module 11 is used to optimize the directed graph based on the target data shape when the target data shape is less than or equal to the maximum data shape, to obtain an optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model;
[0091] The processing module 12 is used to process the target image using the optimized directed graph to obtain the processing result corresponding to the target image.
[0092] The image processing apparatus provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0093] In one embodiment, such as Figure 6 As shown, the optimization module 11 includes: a calling unit 111 and an optimization unit 112; wherein:
[0094] Calling unit 111 is used to call the preset optimization method.
[0095] The optimization unit 112 is used to optimize the directed graph using the optimization method and the target data shape to obtain the optimized directed graph.
[0096] The image processing apparatus provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0097] In one embodiment, the optimization unit 112 is used to optimize the operators and compile time corresponding to each node of the directed graph using the optimization method and the target data shape, so as to obtain the optimized directed graph.
[0098] The image processing apparatus provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0099] In one embodiment, such as Figure 7 The aforementioned device further includes: a first determining module 13; wherein:
[0100] The first determining module 13 is used to determine the maximum data shape based on the attribute information of each image in the image set to be processed; the attribute information includes the size of the image and the resolution of the image.
[0101] The image processing apparatus provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0102] In one embodiment, such as Figure 8 The aforementioned device further includes: a second determining module 14; wherein:
[0103] The second determining module 14 is used to determine the maximum data shape using the constraints corresponding to the neural network model.
[0104] The image processing apparatus provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0105] In one embodiment, such as Figure 9 As shown, the above-mentioned compilation module 10 includes: compilation unit 101, wherein:
[0106] The compilation unit 101 is used to construct the directed graph by taking each layer of the neural network model as nodes and the connection relationship between each layer as edges, according to the maximum data shape.
[0107] The image processing apparatus provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0108] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0109] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0110] Based on the maximum data shape corresponding to the image set to be processed, the preset neural network model is compiled into a directed graph;
[0111] If the target data shape is not larger than the maximum data shape, the directed graph is optimized based on the target data shape to obtain an optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model.
[0112] The optimized directed graph is used to process the target image to obtain the processing result corresponding to the target image.
[0113] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0114] Invoke the preset optimization method;
[0115] The directed graph is optimized using the optimization method and the target data shape to obtain the optimized directed graph.
[0116] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0117] Using the optimization method and the target data shape, the operators and compile time corresponding to each node of the directed graph are optimized to obtain the optimized directed graph.
[0118] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0119] The maximum data shape is determined based on the attribute information of each image in the image set to be processed; the attribute information includes the image size and the image resolution.
[0120] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0121] The maximum data shape is determined using the constraints corresponding to the neural network model.
[0122] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0123] Based on the maximum data shape, the directed graph is constructed by taking each layer of the neural network model as nodes and the connections between the layers as edges.
[0124] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0125] Based on the maximum data shape corresponding to the image set to be processed, the preset neural network model is compiled into a directed graph;
[0126] If the target data shape is not larger than the maximum data shape, the directed graph is optimized based on the target data shape to obtain an optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model.
[0127] The optimized directed graph is used to process the target image to obtain the processing result corresponding to the target image.
[0128] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0129] Invoke the preset optimization method;
[0130] The directed graph is optimized using the optimization method and the target data shape to obtain the optimized directed graph.
[0131] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0132] Using the optimization method and the target data shape, the operators and compile time corresponding to each node of the directed graph are optimized to obtain the optimized directed graph.
[0133] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0134] The maximum data shape is determined based on the attribute information of each image in the image set to be processed; the attribute information includes the image size and the image resolution.
[0135] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0136] The maximum data shape is determined using the constraints corresponding to the neural network model.
[0137] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0138] Based on the maximum data shape, the directed graph is constructed by taking each layer of the neural network model as nodes and the connections between the layers as edges.
[0139] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0140] Based on the maximum data shape corresponding to the image set to be processed, the preset neural network model is compiled into a directed graph;
[0141] If the target data shape is not larger than the maximum data shape, the directed graph is optimized based on the target data shape to obtain an optimized directed graph; the target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model.
[0142] The optimized directed graph is used to process the target image to obtain the processing result corresponding to the target image.
[0143] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0144] Invoke the preset optimization method;
[0145] The directed graph is optimized using the optimization method and the target data shape to obtain the optimized directed graph.
[0146] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0147] Using the optimization method and the target data shape, the operators and compile time corresponding to each node of the directed graph are optimized to obtain the optimized directed graph.
[0148] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0149] The maximum data shape is determined based on the attribute information of each image in the image set to be processed; the attribute information includes the image size and the image resolution.
[0150] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0151] The maximum data shape is determined using the constraints corresponding to the neural network model.
[0152] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0153] Based on the maximum data shape, the directed graph is constructed by taking each layer of the neural network model as nodes and the connections between the layers as edges.
[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0155] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0156] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. An image processing method, characterized in that, The method includes: Based on the maximum data shape corresponding to the image set to be processed, the layers of the preset neural network model are used as nodes, and the connections between the layers are used as edges to construct a directed graph. If the target data shape is not larger than the maximum data shape, a preset optimization method is invoked. Using the optimization method and the target data shape, the operators and compile time corresponding to each node of the directed graph are optimized to obtain the optimized directed graph. The target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model. The optimized directed graph is used to process the target image to obtain the processing result corresponding to the target image.
2. The method according to claim 1, characterized in that, The method further includes: The maximum data shape is determined based on the attribute information of each image in the image set to be processed; the attribute information includes the image size and the image resolution.
3. The method according to claim 2, characterized in that, The method further includes: The mean of the image size and the mean of the image resolution in the image set to be processed are determined as the maximum data shape; Alternatively, the maximum value of each image size and the maximum value of each image resolution in the image set to be processed can be determined as the aforementioned maximum data shape.
4. The method according to claim 1, characterized in that, The method further includes: The maximum data shape is determined using the constraints corresponding to the neural network model.
5. The method according to claim 4, wherein the constraints include at least one of the number of layers in the neural network model, the maximum size of the image that the neural network model can process, and the maximum resolution of the image that the neural network model can process.
6. The method according to any one of claims 1-5, characterized in that, The optimization method is obtained by pre-encapsulating all the operations that need to be performed when the shape of the model's input data is not fixed; the operations that need to be performed include at least one of modifying operator attributes and modifying compile-time optimizations.
7. An image processing apparatus, characterized in that, The device includes: The compilation module is used to construct a directed graph based on the maximum data shape corresponding to the image set to be processed, taking each layer of the preset neural network model as nodes and the connection relationship between the layers as edges. An optimization module is used to call a preset optimization method when the target data shape is less than or equal to the maximum data shape. The optimization method and the target data shape are used to optimize the operators and compilation time corresponding to each node of the directed graph to obtain an optimized directed graph. The target data shape is the data shape corresponding to the target image to be processed in the current stage of the neural network model. The processing module is used to process the target image using the optimized directed graph to obtain the processing result corresponding to the target image.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.